I need to generate unique ID's for objects I'll be creating from DTML. This of course needs to be thread safe. Although I can't do this out of the box from within DTML (that I know of), I could get thread safety using an external method, or a persistant object written in Python, and using locks provided by the Python thread module. I think a persistant object (lets call it ZSequence) would be best. However, all this falls down if we want the code to run under ZEO in the future. And using a SEQUENCE in my Oracle database might work for me, but I want avoid using my RDBMS if possible. Can I use the Transaction system? I don't want to do a full commit, as this would have unknow effects on whatever is referencing this object. Can I commit changes to a dirty object, without affecting the rest of the transaction? (I've looked at cPersistence.c, which did nothing but remind we why I havn't programmed in C for years :-) ) Any thoughts? Have I missed some fundamental ZODB concept that solves these issues? ___ // Zen (alias Stuart Bishop) Work: zen@cs.rmit.edu.au // E N Senior Systems Alchemist Play: zen@shangri-la.dropbear.id.au //__ Computer Science, RMIT WWW: http://www.cs.rmit.edu.au/~zen
Stuart 'Zen' Bishop wrote:
I need to generate unique ID's for objects I'll be creating from DTML. This of course needs to be thread safe. Although I can't do this out of the box from within DTML (that I know of), I could get thread safety using an external method, or a persistant object written in Python, and using locks provided by the Python thread module. I think a persistant object (lets call it ZSequence) would be best.
Have you seen my How-To on generating Unique ids? It may not be _exactly_ what you are looking for, but it will provide a decent starting place. The How-To is in the Zope Site's How-to list located at: http://www.zope.org/Documentation/How-To -- "They laughed at Columbus, they laughed at Fulton, they laughed at the Wright brothers. But they also laughed at Bozo the Clown." -- Carl Sagan
On Sat, 9 Oct 1999, Bill Anderson wrote:
Stuart 'Zen' Bishop wrote:
I need to generate unique ID's for objects I'll be creating from DTML. This of course needs to be thread safe. Although I can't do this out of the box from within DTML (that I know of), I could get thread safety using an external method, or a persistant object written in Python, and using locks provided by the Python thread module. I think a persistant object (lets call it ZSequence) would be best.
Have you seen my How-To on generating Unique ids? It may not be _exactly_ what you are looking for, but it will provide a decent starting place.
But What happens when two ID's are generated at exactly the same time through different threads? I found a gem that I thinks answers my queries at: http://www.zope.org/Documentation/Developer/Models/ZODB/ZODB_Doc.html "When a transaction is to be committed, all of the objects modified by the transaction are checked to see if they have been invalidated. If any objects modified by the transaction have been invalidated, then the transaction is aborted and a ZODB3.ConflictError exception is raised. An application (e.g. the Python Object Publisher) should catch the ZODB3.Conflict exception and attempt to re-execute the transaction." So an object can simply have an integer counter that is incremented as necessary, and if two threads try to modify the counter at about the same time, the one that tries to commit last will rollback and be reexecuted. If this is true, its so trivial a counter class and locking is totally unnecessary :-) And it should work with ZEO as well from what I understand of its architecture. ___ // Zen (alias Stuart Bishop) Work: zen@cs.rmit.edu.au // E N Senior Systems Alchemist Play: zen@shangri-la.dropbear.id.au //__ Computer Science, RMIT WWW: http://www.cs.rmit.edu.au/~zen
Stuart 'Zen' Bishop wrote:
Have you seen my How-To on generating Unique ids? It may not be _exactly_ what you are looking for, but it will provide a decent starting place.
But What happens when two ID's are generated at exactly the same time through different threads?
ZODB will not allow two persistent objects to be overwriten at the same time.
So an object can simply have an integer counter that is incremented as necessary, and if two threads try to modify the counter at about the same time, the one that tries to commit last will rollback and be reexecuted. If this is true, its so trivial a counter class and locking is totally unnecessary :-) And it should work with ZEO as well from what I understand of its architecture.
Yep. Because it's persistent however, every unique id will cause a transaction, in a ZEO environment, this would cause lots of object colisions over the wire. Probably not very scalable. ZODB has a lot of potention for non version-based storages however, we are currently working on an projecy oriented toward just that. -Michel
Excuse the massive quote, but I need all the context:
But What happens when two ID's are generated at exactly the same time through different threads?
I found a gem that I thinks answers my queries at: http://www.zope.org/Documentation/Developer/Models/ZODB/ZODB_Doc.html
"When a transaction is to be committed, all of the objects modified by the transaction are checked to see if they have been invalidated. If any objects modified by the transaction have been invalidated, then the transaction is aborted and a ZODB3.ConflictError exception is raised. An application (e.g. the Python Object Publisher) should catch the ZODB3.Conflict exception and attempt to re-execute the transaction."
So an object can simply have an integer counter that is incremented as necessary, and if two threads try to modify the counter at about the same time, the one that tries to commit last will rollback and be reexecuted. If this is true, its so trivial a counter class and locking is totally unnecessary :-) And it should work with ZEO as well from what I understand of its architecture.
Okay, so a method which tries to maintain sub-objects with incrementing ids could do something like this: def xmlrpc_manage_addNi(self, title=''): """Anyone ever noticed that without a doc string, Zope complains LOTS?""" try: while 1: self.lastNi = self.lastNi + 1 id = str(self.lastNi) if hasattr(self, id): # there's someone else in the system! continue v = self._checkId(id) # if someone slips in here, it'll still break :/ break object = Ni(id, title) if v is not None: id = v try: t = object.meta_type except: t = None self._objects = self._objects + ({ 'id': id, 'meta_type':t},) self._setOb(id,object) object = self._getOb(id) object.manage_afterAdd(object, self) I haven't tried using multi-threading to force the race condition yet, but I suspect that it's still possible for two calls to concurrently march through the method and still try and commit the same id. How the hell do you catch the ZODB3.ConflictError if this happens? I guess I'm not sold on the whole "the one that tries to commit last will rollback and be reexecuted" thing. Regards, Garth. -- <gtk@well.com>
Stuart 'Zen' Bishop wrote:
I need to generate unique ID's for objects I'll be creating from DTML. This of course needs to be thread safe. Although I can't do this out of the box from within DTML (that I know of), I could get thread safety using an external method, or a persistant object written in Python, and using locks provided by the Python thread module. I think a persistant object (lets call it ZSequence) would be best.
However, all this falls down if we want the code to run under ZEO in the future. And using a SEQUENCE in my Oracle database might work for me, but I want avoid using my RDBMS if possible.
Can I use the Transaction system? I don't want to do a full commit, as this would have unknow effects on whatever is referencing this object. Can I commit changes to a dirty object, without affecting the rest of the transaction? (I've looked at cPersistence.c, which did nothing but remind we why I havn't programmed in C for years :-) )
Any thoughts? Have I missed some fundamental ZODB concept that solves these issues?
No. No one has had the need to solve the problem yet concerning the ZEO issue. For a single process space Zope (ie, intra-thread but inter-process) there is a how-to on how to generate unique ides. For a zeo envorinment, I would suggesting using Apache's unique ID module which passes in a unique id with an environment variable for each request, guarenteed (or at least odds in the billions to one against it not being) unique across a set of machines which synchronize via Network Time Protocol. -Michel
___ // Zen (alias Stuart Bishop) Work: zen@cs.rmit.edu.au // E N Senior Systems Alchemist Play: zen@shangri-la.dropbear.id.au //__ Computer Science, RMIT WWW: http://www.cs.rmit.edu.au/~zen
_______________________________________________ Zope maillist - Zope@zope.org http://www.zope.org/mailman/listinfo/zope
(Related lists - please, no cross posts or HTML encoding!
To receive general Zope announcements, see: http://www.zope.org/mailman/listinfo/zope-announce
For developer-specific issues, zope-dev@zope.org - http://www.zope.org/mailman/listinfo/zope-dev )
On Sun, 10 Oct 1999, Michel Pelletier wrote:
Stuart 'Zen' Bishop wrote:
I need to generate unique ID's for objects I'll be creating from DTML. This
request, guarenteed (or at least odds in the billions to one against it not being) unique across a set of machines which synchronize via Network Time Protocol.
-Michel
There are a lot of ways of doing this and of course it depends on the application. Once when I needed to do something similar I generated an md5 checksum out of a property of the Zope object and its parent folder id. In my application such a combination was unique. If you have some property of your object (in combination with some other info perhaps) that is quaranteed to be unique you can use the md5 module that comes standard with python to generate a unique id. Pavlos
I need to generate unique ID's for objects I'll be creating from DTML. This
application. Once when I needed to do something similar I generated an md5 checksum out of a property of the Zope object and its parent folder id. In my application such a combination was unique. If you have
I just tried this out, creating md5er.py in my extensions directory: # md5er external method module for Zope def md5(inputstring): import md5 m=md5.new() m.update(inputstring) return m.digest() ... then creating external method md5er aimed at the module md5er, function name md5. So far, not bad, since it's my first Zope external method and also the first Python code I've ever written. I like this stuff. Nice and simple. But I digress. <dtml-var expr="md5er('ggdgfg')"> ... works just as expected. The output is 8-bit garbage, though -- anyone know of a nice unpack or hex conversion function I can call to clean that up into a clean string? Here's something wierd, though: <dtml-var expr="md5er(id)"> ... gives me an error: Error Type: TypeError Error Value: read-only buffer, Python Method Any suggestions?
On Mon, 11 Oct 1999, gtk wrote:
Here's something wierd, though:
<dtml-var expr="md5er(id)">
This is one of Zope's idiosyncrasies. id is ctually a metho when called in this manner. <dtml-var expr="md5er(id())"> should work you can use module binascii to convert it to base64 like import binascii binascii.b2a_base64(a.digest())[:-1] this will return a base64 representation of your digest, and it will also strip the trailing newline. Pavlos
On Mon, 11 Oct 1999, gtk wrote:
But I digress.
<dtml-var expr="md5er('ggdgfg')">
... works just as expected. The output is 8-bit garbage, though -- anyone know of a nice unpack or hex conversion function I can call to clean that up into a clean string?
This one was suggested by Guido few years ago: s = m.digest() return "%02x"*len(s) % tuple(map(ord, s)) Oleg. ---- Oleg Broytmann National Research Surgery Centre http://sun.med.ru/~phd/ Programmers don't die, they just GOSUB without RETURN.
gtk wrote:
Here's something wierd, though:
<dtml-var expr="md5er(id)">
... gives me an error:
Error Type: TypeError Error Value: read-only buffer, Python Method
Any suggestions?
'id' is sometimes an attribute, sometimes a method, and sometimes a 'computed attribute'. In this case, you need to call 'id' with 'id()'. -Michel
participants (6)
-
Bill Anderson -
gtk -
Michel Pelletier -
Oleg Broytmann -
Pavlos Christoforou -
Stuart 'Zen' Bishop