[ZODB-Dev] notes re trying to run ZODB on PyPy

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/24/2011 08:02 PM, David Glick wrote:
> Alan Runyan asked me to post my notes from my attempt to get ZODB 
> running on PyPy. It's very much an experimental work in progress
> (that I got distracted from), but hopefully this is at least useful
> to anyone else who wants to attempt the same thing.
> 
> I first tried building ZODB trunk (with C extensions). Ran into the 
> following issues: * PyPy is missing an implementation of the
> _Py_ForgetReference macro which is used by ZODB's
> persistent/cPickleCache.c * The check_argument_cmp check in
> BTrees/objectkeymacros.h fails for the root object of the database on
> PyPy * I got a PyPy exception re wrap_objobjargspec which appeared to
> be some issue with using the __setitem__ slot of an extension type
> 
> At this point I gave up on trying to build the C extensions, and
> instead did the following: 1. Started with Jim's python-btrees branch
>  (http://svn.zope.org/repos/main/ZODB/branches/jim-python-btrees) 2.
> Replaced the 'persistent' package with a checkout of Tres' 
> pure-Python persistent implementation 
> (http://svn.zope.org/repos/main/persistent/trunk/persistent) 3. Set
> exts = [] in setup.py to disable all C extensions when installing.
> 
> Next I worked around some issues in the pure-Python branches: *
> BTrees.___BTree.fsBucket class was missing toString and fromString 
> methods, which I implemented. * persistent.picklecache.PickleCache
> class was missing update_object_size_estimation method (and indeed
> the whole limit-cache-size-by-bytes feature). I added it as a no-op
> stub. * persistent.picklecache.PickleCache.__setitem__ was raising a
> KeyError for a duplicate oid even when trying to set the same object
> already stored under that key * persistent.TimeStamp did not give the
> Python implementation, which I fixed by an import of
> persistent.timestamp + module alias if persistent.TimeStamp (the C
> module) is missing * persistent.pyPersistence.Persistent's __new__
> needs to accept *args and **kw
> 
> At this point I was able to start up the Pyramid ZODB scaffold and
> add an object to the DB root.
> 
> Next I tried to get zodbshootout running as a benchmark. I didn't get
>  this working, as I got stuck on an issue with the pure-Python 
> PersistentMapping getting committed without its object state
> (appeared to be something to do with failing to get unghosted before
> it gets committed). Along the way, I noticed a couple
> incompatibilities with PyPy because PyPy implements the cPickle
> module as an alias to the pure-Python pickle module. This leads to
> some problems due to slight semantic differences between cPickle and
> pickle. In particular, cPickle.Pickler accepts a lone protocol
> argument; pickle.Pickler does not (leading to an error in
> ZEO.zrpc.marshal.encode). And pickle.Unpickler does not have a noload
> method, which ZODB.serialize.referencesf uses to obtain the
> referenced oids from a pickle without the overhead of unpickling.
> 
> That's as far as I got. Hope this is helpful to the next guy. :)

Cool, great work.  Are the changes you needed to make available as
patchers, or checked in?



Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk5/RgAACgkQ+gerLs4ltQ7NmgCg2n4GT1wQWxPaRNWCguOtAb8q
OQ4AoKcbDGep0ig5ZL/x+/m4hNeAVyuq
=KtIZ
-----END PGP SIGNATURE-----