[ZODB-Dev] notes re trying to run ZODB on PyPy

[David Glick]

On 9/25/11 8:17 AM, Tres Seaver wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 09/24/2011 08:02 PM, David Glick wrote:
>> Alan Runyan asked me to post my notes from my attempt to get ZODB
>> running on PyPy. It's very much an experimental work in progress
>> (that I got distracted from), but hopefully this is at least useful
>> to anyone else who wants to attempt the same thing.
>>
>> I first tried building ZODB trunk (with C extensions). Ran into the
>> following issues: * PyPy is missing an implementation of the
>> _Py_ForgetReference macro which is used by ZODB's
>> persistent/cPickleCache.c * The check_argument_cmp check in
>> BTrees/objectkeymacros.h fails for the root object of the database on
>> PyPy * I got a PyPy exception re wrap_objobjargspec which appeared to
>> be some issue with using the __setitem__ slot of an extension type
>>
>> At this point I gave up on trying to build the C extensions, and
>> instead did the following: 1. Started with Jim's python-btrees branch
>>   (http://svn.zope.org/repos/main/ZODB/branches/jim-python-btrees) 2.
>> Replaced the 'persistent' package with a checkout of Tres'
>> pure-Python persistent implementation
>> (http://svn.zope.org/repos/main/persistent/trunk/persistent) 3. Set
>> exts = [] in setup.py to disable all C extensions when installing.
>>
>> Next I worked around some issues in the pure-Python branches: *
>> BTrees.___BTree.fsBucket class was missing toString and fromString
>> methods, which I implemented. * persistent.picklecache.PickleCache
>> class was missing update_object_size_estimation method (and indeed
>> the whole limit-cache-size-by-bytes feature). I added it as a no-op
>> stub. * persistent.picklecache.PickleCache.__setitem__ was raising a
>> KeyError for a duplicate oid even when trying to set the same object
>> already stored under that key * persistent.TimeStamp did not give the
>> Python implementation, which I fixed by an import of
>> persistent.timestamp + module alias if persistent.TimeStamp (the C
>> module) is missing * persistent.pyPersistence.Persistent's __new__
>> needs to accept *args and **kw
>>
>> At this point I was able to start up the Pyramid ZODB scaffold and
>> add an object to the DB root.
>>
>> Next I tried to get zodbshootout running as a benchmark. I didn't get
>>   this working, as I got stuck on an issue with the pure-Python
>> PersistentMapping getting committed without its object state
>> (appeared to be something to do with failing to get unghosted before
>> it gets committed). Along the way, I noticed a couple
>> incompatibilities with PyPy because PyPy implements the cPickle
>> module as an alias to the pure-Python pickle module. This leads to
>> some problems due to slight semantic differences between cPickle and
>> pickle. In particular, cPickle.Pickler accepts a lone protocol
>> argument; pickle.Pickler does not (leading to an error in
>> ZEO.zrpc.marshal.encode). And pickle.Unpickler does not have a noload
>> method, which ZODB.serialize.referencesf uses to obtain the
>> referenced oids from a pickle without the overhead of unpickling.
>>
>> That's as far as I got. Hope this is helpful to the next guy. :)
> Cool, great work.  Are the changes you needed to make available as
> patchers, or checked in?
>
Most were temporary hacks and the rest have no tests yet, so not checked 
in anywhere. But here's (attached) the diff from my working directory.



----------		
David Glick
 Web Developer
 davidglick at groundwire.org
 206.286.1235x32

Online tools and strategies for the environmental movement. 

Sign up for our newsletter: http://www.groundwire.org/email-capture


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ZODB_pypy.patch
Url: http://mail.zope.org/pipermail/zodb-dev/attachments/20110925/b0d7c656/attachment.pl