[ZODB-Dev] ZODB and Parallel-Python
Flavio Coelho
fccoelho at gmail.com
Tue Feb 19 12:03:04 EST 2008
Hi,
I am trying to use PP with ZODB.
Parallel Python (www.parallelpython.com) provides works as a job
server which starts processes, thus providing a mechanism for the
parallel execution of python code on SMP machines or clusters. Each
job basically calls a function and then executes it (module
dependencies are passed as an import list for each job). For
concurrent writing to the DB I had to find a way to have each of these
processes create their own connections. However, PP, passes variables
to the job-processes as pickles, and the db object is not picklable so
it cannot be passed to PP jobs.
The solution I am trying is to offload the db handling code to a
separate module that can be added to the import list of a PP job:
dbserve.py
Code:
import ZODB
import ZEO.ClientStorage
class Data:
def __init__(self):
#setting up ZEO server
#os.system('runzeo -a localhost:8008 -f SN.fs &')
# setup database
storage = ZEO.ClientStorage.ClientStorage(('localhost',8008))
self.db = ZODB.DB(storage)
def getConn(self):
conn = self.db.open()
return conn.root()
That seems to be a solution, but fails because now each process tries
to create a ClientStorage object from which to derive a connection.
and the server won't accept it....
An error has occured during the function execution
Traceback (most recent call last):
File "/usr/lib/python2.5/site-packages/ppworker.py", line 91, in run
__result = __f(*__args)
File "<string>", line 4, in cenario
File "<string>", line 10, in Reps
File "/home/flavio/Documents/Papers/VacBelief/src/dbserve.py", line
14, in __init__
self.db = ZODB.DB(storage)
File "/usr/lib/python2.5/site-packages/ZODB3-3.8.0-py2.5-linux-i686.egg/ZODB/DB.py",
line 270, in __init__
storage.load(z64, '')
File "/usr/lib/python2.5/site-packages/ZEO/ClientStorage.py", line
757, in load
p, s, v, pv, sv = self._server.zeoLoad(oid)
File "/usr/lib/python2.5/site-packages/ZEO/ServerStub.py", line 82, in zeoLoad
return self.rpc.call('zeoLoad', oid)
File "/usr/lib/python2.5/site-packages/ZEO/zrpc/connection.py", line
484, in call
r_flags, r_args = self.wait(msgid)
File "/usr/lib/python2.5/site-packages/ZEO/zrpc/connection.py", line
574, in wait
raise DisconnectedError()
Anyone has any experience with ZODB and PP? any sugestions?
In summary: I need to expose a DB Object to multiple processes so that
they can derive their own connections But this object cannot passed at
job-creation time because they are not pickleable, and they apparently
cannot be created by each job because I cannot have multiple
ClientStorage objects targetting the same phisical zeo server (is this
a real limitation?)
thanks in advance,
Flávio C. Coelho
More information about the ZODB-Dev
mailing list