November 2001 - Zope

RE: [Zope] zope operations atomic?
by Clark OBrien 02 Nov '01

02 Nov '01

This thread is getting hard to follow. The second block below that you attribute to me was actually Kapil's response to something I had said. -----Original Message----- From: Chris McDonough [mailto:chrism@zope.com] Sent: Thursday, November 01, 2001 4:50 PM To: Kapil Thangavelu; Clark OBrien Cc: zope(a)zope.org Subject: Re: [Zope] zope operations atomic? Clark wrote: > > suppose I have a method AtoB that changes every > > letter A to B. > > suppose I have another method BtoC that changes the > > leter B to C. > > > > what is the outcome of invoking these methods in > > different thread contexts > > on the > > string ABABABAB? > > Kapil replied: > it depends on the concurrency. if both methods execute > concurrently, than one will conflict and one will > commit from trying to write on the same object. which > one is hard to determine apriori... i'm taking a wild > guess that it would be first one to commit. Chris, > could you comment on this? I'm not totally sure I understand the question (was it an allegory?) but a literal response is: If there is a conflict created by a first-time concurrent run of the two methods, the string AAAAAAAA will be modified to become: BBBBBBBB if AtoB wins the commit or ACACACAC if BtoC wins the commit The "loser" of the commit would on a retry subsequently cause the string to become: CCCCCCCC (if BtoC was the loser) or BCBCBCBC (if AtoB was the loser) (crossing fingers ;-) Clark said: > > >your joking, right... besides the fact that > > excessive locking kills > > >performance and scalability, i *know* i don't want > > to have to litter my > > code > > >with locks and thinking about concurrency. the zodb > > greatly mitigates > > >concurrency issues, making them almost non-existent > > for most code. moving > > >from this paradigm to some application level > > locking paradigm greatly > > >complicates code and would make it damm near > > impossible to be productive > > with > > >zope, IMO. I sorta feel like I'm answering brainteasers at this point. Can I help somebody solve an actual problem here? ;-) - C

1 0

add ons for zope
by jimmy coyne 02 Nov '01

02 Nov '01

Hi all, Is there a standard way to install add ons for zope ? I'm asking this because I download the InternationalObjects which only has two files( __init__.py and __init__.pyc) and no instruction on what to do with them .Has any body installed any thing similar . thanks in advance Jim

2 1

DocumentLibrary 1.0b3 / Zope 2.4.2 (newbie?) problem
by Kolb, Hap 02 Nov '01

02 Nov '01

Hi, any attempt to submit a document to my newly created document library under zope 2.4.2 yields the following error (traceback appended below): Error Type: NameError Error Value: global name 'ZopeTime' is not defined Now that's an error so basic that I suspect that it's me who is doing something wrong---if I only knew what...! Any help would be highly appreciated! Best, ...hap Traceback (innermost last): File /data1/software/Zope/zope-2.4.2/lib/python/ZPublisher/Publish.py, line 223, in publish_module File /data1/software/Zope/zope-2.4.2/lib/python/ZPublisher/Publish.py, line 187, in publish File /data1/software/Zope/zope-2.4.2/lib/python/Zope/__init__.py, line 226, in zpublisher_exception_hook (Object: library) File /data1/software/Zope/zope-2.4.2/lib/python/ZPublisher/Publish.py, line 171, in publish File /data1/software/Zope/zope-2.4.2/lib/python/ZPublisher/mapply.py, line 160, in mapply (Object: document_submit) File /data1/software/Zope/zope-2.4.2/lib/python/ZPublisher/Publish.py, line 112, in call_object (Object: document_submit) File /data1/software/Zope/zope-2.4.2/lib/python/OFS/DTMLMethod.py, line 195, in __call__ (Object: document_submit) File /data1/software/Zope/zope-2.4.2/lib/python/DocumentTemplate/DT_String.py, line 546, in __call__ (Object: document_submit) File /data1/software/Zope/zope-2.4.2/lib/python/DocumentTemplate/DT_Try.py, line 212, in render File /data1/software/Zope/zope-2.4.2/lib/python/DocumentTemplate/DT_Try.py, line 221, in render_try_except File /data1/software/Zope/zope-2.4.2/lib/python/DocumentTemplate/DT_With.py, line 148, in render (Object: Documents) File /data1/software/Zope/zope-2.4.2/lib/python/DocumentTemplate/DT_With.py, line 133, in render (Object: addDocumentFile(REQUEST)) File /data1/software/Zope/zope-2.4.2/lib/python/DocumentTemplate/DT_Util.py, line 231, in eval (Object: addDocumentFile(REQUEST)) (Info: REQUEST) File <string>, line 2, in f File /data1/software/Zope/zope-2.4.2/Products/DocumentLibrary/DocumentStore.py, line 355, in addDocumentFile (Object: Documents) File /data1/software/Zope/zope-2.4.2/lib/python/OFS/ObjectManager.py, line 324, in _setObject (Object: Documents) File /data1/software/Zope/zope-2.4.2/lib/python/Products/ZCatalog/CatalogAwarenes s.py, line 114, in manage_afterAdd (Object: 1004694129.27) File /data1/software/Zope/zope-2.4.2/lib/python/Products/ZCatalog/CatalogAwarenes s.py, line 184, in index_object (Object: 1004694129.27) File /data1/software/Zope/zope-2.4.2/Products/DocumentLibrary/CatalogPlus.py, line 122, in catalog_object File /data1/software/Zope/zope-2.4.2/lib/python/Products/ZCatalog/Catalog.py, line 382, in catalogObject File /data1/software/Zope/zope-2.4.2/lib/python/Products/ZCatalog/Catalog.py, line 495, in recordify File /data1/software/Zope/zope-2.4.2/lib/python/OFS/DTMLMethod.py, line 188, in __call__ (Object: date) File /data1/software/Zope/zope-2.4.2/lib/python/DocumentTemplate/DT_String.py, line 546, in __call__ (Object: date) File /data1/software/Zope/zope-2.4.2/lib/python/DocumentTemplate/DT_Util.py, line 231, in eval (Object: ZopeTime().Date()) (Info: ZopeTime) File <string>, line 2, in f (Object: guarded_getattr) NameError: (see above)

1 0

Assigning One dtml-var to Another
by Ben Ocean 02 Nov '01

02 Nov '01

Hi; I have a dtml-var called *select-blueline* It reads thus: <dtml-var expr="_.whrandom.choice(images.bluelines.objectValues(['DTML Method']))"> I need to be able to assign the value generated from this method to ANOTHER method that I can call. In other words... <dtml-var x> = <dtml-var select-blueline> Then later call <dtml-var x> How do I do this? TIA, BenO

6 6

REQUEST.items() problem
by Juan Garcia Garcia 02 Nov '01

02 Nov '01

Hi, i'm trying to show all data contained by REQUEST, but when i write the next code, the computer overloads and Zope doesn't response: <dtml-in "REQUEST.items()" sort> <dtml-var sequence-key>:<dtml-var sequence-item><BR> </dtml-in> If i use REQUEST.form.items() or REQUEST.cookies.items(), it works perfectly, but there is data in 'Other' that i need. Am i doing something wrong or is a problem of Zope? I'm using Zope 2.3.3, python 1.5.2 and Debian/woody. Thanks. Bye.

4 7

Re: [Zope] RE: CMF Newbie Presentation and Viewlet Info
by Stephan Goeldi 02 Nov '01

02 Nov '01

>And me prevents this funny technology from viewing the application because >I do not want to download any binary plugin. I thought that it does not need any plugin because it depents on a Java applet. When a java applet is a plugin for you ... >You will say: Just do not use such stupid browsers. I will never say something like this, especially because I am trying to be a usability designer. Usability design is done for humans and not for browsers. When the Viewlet approach sucks for a lot of people, then I have to look for another solution. But you won't tell me that e.g. flash would be a solution, would you? _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp

4 3

action modifier adds calling method to URL?
by Milos Prudek 02 Nov '01

02 Nov '01

I have a method "run" that is called from method "display" on /presley/elvis folder. When using simple form as follows... <form action="run"> <input type="submit" name="submit" value="Run!"> </form> ... it goes to a correct URL: /presley/elvis/run ... and URL0 is /presley/elvis/run However, I'd like to use :action modifier, as follows: <form action=""> <input type="submit" name="run:action" value="Run!"> <input type="submit" name="set:action" value="Set!> </form> This works fine, but it goes to: /presley/elvis/display/run or /presley/elvis/display/set. and URL0 is also /presley/elvis/display/run or /presley/elvis/display/set. This '/display/' disturbs logic of my program (things like PARENTS). Is this standard behaviour, or a Zope fault? If it's standard, what is the advantage of this behaviour? -- Milos Prudek

2 2

RE: [Zope] zope operations atomic?
by Kapil Thangavelu 02 Nov '01

02 Nov '01

--- Clark OBrien <COBrien(a)isis-server.vuse.vanderbilt.edu> wrote: > > > > > Thanks for the clarification - more comments below. > clark > no problem. one clarification to my clarification:), zope threads do not maintain persistent connections to the zodb, they open a new connection for every request, the zodb internally stores connection objects (and associated caches) in a LIFO stack to maximize cache hits. i'm actually a little unclear when the these connections are synced up with the db (cache invalidated). > >each thread maintains its own connection to the > zodb, and each connection > >mantains its own cache. when you load up on object > you are not deep > loading, > >your loading up all of that object's data minus any > persistent attributes > it > >might have. so in a given folder stuffed with > standard zope objects you > only > >load up that folders attributes unless you > explicitly access the other > >peristent objects. objects get stuffed into cache > after access, for faster > >lookups. as chris mentioned zope treats every web > request as a transaction. > > >when a thread tries to commit an object it, the db > checks to see if it has > >been updated since the transaction began, in which > case you get a conflict. > > >so what does atomic mean, that a given set of > operations either succeeds as > a > >whole or fails. the zodb passes this standard. the > issue you seem to layout > > >below is that your more comfortable with explicit > locking... > > atomic implies a set of opperations succeed or they > have no effect on the > data. > > > suppose I have a method AtoB that changes every > letter A to B. > suppose I have another method BtoC that changes the > leter B to C. > > what is the outcome of invoking these methods in > different thread contexts > on the > string ABABABAB? > it depends on the concurrency. if both methods execute concurrently, than one will conflict and one will commit from trying to write on the same object. which one is hard to determine apriori... i'm taking a wild guess that it would be first one to commit. Chris, could you comment on this? but what does this mean in terms of atomic operation? independently running these methods in different contexts implies that these each forms a separate atomic operation with the additional set of operations in each context respectively. so each operation will either succeed or fail, if their is no consistency check made on the value of the string, than its possible to achieve different values depending on timing. the "loser" will retry, but in either case the operations are still atomic in that they succeed or fail. to take a page from chris's rdbms analogy. an rdbms app might get around this by explicitly locking the row. so you could know the value of the string assuming you know the timings of the threads, and which one gets to the lock first. but unless you define a consistency check, you're at the mercy of the thread timings, which is a poor way to determine consistency for concurrent operations. > > >your joking, right... besides the fact that > excessive locking kills > >performance and scalability, i *know* i don't want > to have to litter my > code > >with locks and thinking about concurrency. the zodb > greatly mitigates > >concurrency issues, making them almost non-existent > for most code. moving > >from this paradigm to some application level > locking paradigm greatly > >complicates code and would make it damm near > impossible to be productive > with > >zope, IMO. > > No I am not joking, resolving conflicts after each > request by simply > rolling back is easy to implement. > The problems with concurrent access are with finding > *intelligent* solutions > not easy solutions. The zope guys were smart and > realized that it is only > rarely > that a web application has to deal with concurrent > access and when this > happens > it should be done at the db layer- as the word > "optimistic" implies. concurrent write access that is. as i said previously in the unquoted material that made up my previous reply, if you're application has a lot of concurrent write operations, i would store the data in an rdbms. > also making manage_changeProperties atomic would be > an the level of C making > strcpy threadsafe. unless you are doing embedded > systems you appreciate such > features. lost me here. cheers kapil > > > -----Original Message----- > > From: kapil thangavelu > [mailto:kthangavelu@earthlink.net] > > Sent: Wednesday, October 31, 2001 3:03 AM > > To: Clark OBrien; 'zope(a)zope.org' > > Subject: Re: [Zope] zope operations atomic? > > > > On Wednesday 31 October 2001 07:11 am, Clark > OBrien wrote: > > > Hi: > > > I cannot find documentation that would tell me > how zope manages thread > > > synchronization > > > > nutshell version. > > > > zope uses optimistic conflict resolution, no > application level locking > > takes > > place, because all threads have their own object > spaces. only on > > commits, is > > there some synchronization (deep within the zodb) > to insure that there > > weren't conflicts, if so a conflict error is > raised, and the request > > gets > > retried. applications can define their own > conflict resolution, see the > > zodb > > wikis and zope developer's guide for more details. > > > > kapil > > > > _______________________________________________ > > Zope maillist - Zope(a)zope.org > > http://lists.zope.org/mailman/listinfo/zope > > ** No cross posts or HTML encoding! ** > > (Related lists - > > > http://lists.zope.org/mailman/listinfo/zope-announce > > http://lists.zope.org/mailman/listinfo/zope-dev ) __________________________________________________ Do You Yahoo!? Find a job, post your resume. http://careers.yahoo.com

2 3

a manage_workspace of my own
by Jerry McRae 02 Nov '01

02 Nov '01

Hi zope, My application uses lots of different types of tables. What I would really like is a custom public manage_workspace interface to my folders. I want the cut/copy/paste and 'add' buttons. I just need the options to: 1) set my own tabs 2) create my own 'types' for the pull-down add box 3) add a column or two 4) limit and/or sort the contents I have looked all over and I cannot figure out how to inherit the manage_workspace logic with a different name and parameters. This seems an obvious interface for certain applications, but no one has mentioned it in the list or the how-to's. Being good with logic, average with Python, and abeginning Zope programmer, where would I even start? --Jerry Zope 2.4.1 ( ) / History repeats itself, and that's one of the things that's wrong with history. --Clarence Darrow

1 1

RE: [Zope] zope operations atomic?
by Clark OBrien 02 Nov '01

02 Nov '01

I will convert my example to code and post the results. I did not understand case b). In particular what does "ghost out" mean. -----Original Message----- From: Chris McDonough [mailto:chrism@digicool.com] Sent: Thursday, November 01, 2001 2:01 PM To: Clark OBrien Cc: 'k_vertigo(a)yahoo.com'; 'zope(a)zope.org' Subject: Re: [Zope] zope operations atomic? Clark OBrien wrote: > It is what happens with this retrying that interests me. As the example > given > below shows, this retry may never succeed. Thus long running operations may > never complete and are effectively starved by faster running operations. > > > > Assume in Zope I have a folder structure like this. > > Folder-1 > -Folder-2 > -Folder-3 > .... > .. > Folder-1000 > > > Suppose I have a script traverseFolder(root) that starts at a given root and > traverses sub-folders adding the attribute foo. > > If I continuously call traversefolder(Folder-1000) how could > traverseFolder(Folder-1) ever complete. I mean, by the time > traverseFolder(Folder-1) could complete traverseFolder(Folder-1000) would > have already committed several times. What would happen with the request > that started traverseFolder(Folder-1), would it keep being retried ad > infinitum. > It is basically starved out of contention by a faster running operations. It does get interesting when you consider that the longer-running transaction might always tend to lose on read conflicts because: a) read conflicts can't be resolved. b) the longer-running transaction *might* "ghost out" Folder-1000 to conserve RAM during the traversal after the first (aborted) commit. If b) is true in your example for every run of traverseFolder(Folder-1), your contention that it will never commit might prove correct. It'd be slightly interesting to try it to see what the behavior actually is. Would you be willing to do so? In any case, we have an answer to this problem if you're willing to lose a bit of consistency. CoreSessionTracking's LowConflictConnection class comes in handy here. If your application is very write-intensive and you've carefully coded a massive hotspot ala your example into it, you can turn off read conflicts by using this class at the expense of some consistency. With read conflicts turned off, I'm positive your example will eventually resolve itself. Maybe it'll take a few minutes of retries, but it will eventually finish. I say that because the longer running of the two scripts will eventually be able to do the commit because it's statistically as likely to "win" a commit as the shorter-running script when there's a write conflict; it just has fewer opportunities to do so. And actually if you used a special FooFolder instead of a Folder for this demonstration, the only thing that changed was foo, and the value of foo was often the same in both connections on the object upon which the transaction conflicted, you could resolve most of the conflicts that could potentially occur here (save for the read conflicts) by giving it a _p_resolveConflict that looked something like this: class FooFolder: def _p_resolveConflict(self, old, saved, new): marker = [] for k,v in saved.items(): if new.get(k, marker) != v: return None for k,v in new.items(): if saved.get(k, marker) != v: return None return new However, the real answer is: Dont design your application like this if you can help it. This is not a good pattern. It's best to avoid hotspots like Folder-1000 in your example. It's no different than continually beating the snot out of a field in a row in a relational database table with writes from multiple threads, where one of the threads is running an overnight transaction and the others are just incrementing the field every second. You have the same consistency, contention, and timeliness issues there, AFAICT, except that it's expressed in terms of pages, locks, and dirty reads. (I'm sure there's an Oracle person waiting around to "WRONG!" me to death, however. ;-) - C

3 3