pickling limitation in external methods?
back in january, i asked a question about this, and jim fulton started to answer. unfortunately, time pressures were such that i had to abandon that line of development for a while. well, i'm back now, and with more information! can somebody tell my why i might be seeing a problem in using pickles in my external methods? i have an external method which uses the pickle module to store and retrieve instances of DateTime (from the mxDateTime module). the mxDateTime module imports fine, but no longer seems to cooperate with the pickler. that is, the external method works fine until it tries to unpickle a DateTime object, whereupon it crashes (with one of the least helpful tracebacks in recent memory): ----- Traceback (innermost last): File C:\Program Files\WebSite\lib\python\ZPublisher\Publish.py, line 877, in publish_module File C:\Program Files\WebSite\lib\python\ZPublisher\Publish.py, line 590, in publish (Info: /testMethod) File C:\Program Files\WebSite\lib\python\Products\ExternalMethod\ExternalMethod.py, line 254, in __call__ (Object: CopySource) (Info: ((), {}, None)) TypeError: read-only buffer, class ----- here's the code itself: ----- def test(): d = dbConnection.dbConnection() r = d.execAndGet("select serial, startDate from myData where state='started'") return '<html><body>%s</body></html>' % (r,) ----- the dbConnection module proxies a database connection. internally, it processes results as pickled python objects. i've isolated the trouble to the DateTime objects. in the execAndGet call above, everything's fine if the result set doesn't include a DateTime object (if i remove the "startDate" from the query). if the result set has a DateTime object, it chokes. they pickle and unpickle just fine and dandy under "normal" python, but not inside an external method. we make heavy use of these objects (and the pickling code) in our backend processing. i've tried replacing the cPickle module that comes with ZOPE with the one i have in my standard python installation, and it makes no difference. is there anything anyone can suggest to make this work? would writing a product (or factory, i can't remember which) be a possible solution?
Wolf Logan wrote:
back in january, i asked a question about this, and jim fulton started to answer. unfortunately, time pressures were such that i had to abandon that line of development for a while. well, i'm back now, and with more information!
can somebody tell my why i might be seeing a problem in using pickles in my external methods? i have an external method which uses the pickle module to store and retrieve instances of DateTime (from the mxDateTime module). the mxDateTime module imports fine, but no longer seems to cooperate with the pickler. that is, the external method works fine until it tries to unpickle a DateTime object, whereupon it crashes (with one of the least helpful tracebacks in recent memory): ----- Traceback (innermost last): File C:\Program Files\WebSite\lib\python\ZPublisher\Publish.py, line 877, in publish_module File C:\Program Files\WebSite\lib\python\ZPublisher\Publish.py, line 590, in publish (Info: /testMethod) File C:\Program Files\WebSite\lib\python\Products\ExternalMethod\ExternalMethod.py, line 254, in __call__ (Object: CopySource) (Info: ((), {}, None)) TypeError: read-only buffer, class -----
This is probably due to a bug in the ExternalMethod error handling. It assumes that the error type is a string, when it is really a class. Check the indicated source line. This will be fixed in the next release.
here's the code itself: ----- def test(): d = dbConnection.dbConnection() r = d.execAndGet("select serial, startDate from myData where state='started'") return '<html><body>%s</body></html>' % (r,) -----
the dbConnection module proxies a database connection. internally, it processes results as pickled python objects.
i've isolated the trouble to the DateTime objects. in the execAndGet call above, everything's fine if the result set doesn't include a DateTime object (if i remove the "startDate" from the query). if the result set has a DateTime object, it chokes. they pickle and unpickle just fine and dandy under "normal" python, but not inside an external method.
we make heavy use of these objects (and the pickling code) in our backend processing. i've tried replacing the cPickle module that comes with ZOPE with the one i have in my standard python installation, and it makes no difference. is there anything anyone can suggest to make this work? would writing a product (or factory, i can't remember which) be a possible solution?
Are mxDateTime objects picklable? If you create one and bind it to 'd', does the following code work: cPickle.loads(cPickle.dumps(d)) does it return a copy of d? Jim P.S. There is an issue with pickling instances of classes defined in external method files. These files are not real modules and can't work with the pickling machinery. This may change in the future. -- Jim Fulton mailto:jim@digicool.com Technical Director (540) 371-6909 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
----- Original Message ----- From: Jim Fulton <jim@digicool.com> Sent: Tuesday, May 04, 1999 4:53 AM
TypeError: read-only buffer, class -----
This is probably due to a bug in the ExternalMethod error handling. It assumes that the error type is a string, when it is really a class. Check the indicated source line. This will be fixed in the next release.
i suspected so. thanks.
Are mxDateTime objects picklable?
If you create one and bind it to 'd', does the following code work:
cPickle.loads(cPickle.dumps(d))
does it return a copy of d?
yup. they are picklable objects in general. the mxDateTime package registers the types so that the pickle and cPickle modules can handle them properly. that's why the code works when run under "raw" python, and why i suspected an incompatibility with the cPickle module that comes with win32 ZOPE. but replacing that module with one that is known to work with mxDateTime didn't help.
P.S. There is an issue with pickling instances of classes defined in external method files. These files are not real modules and can't work with the pickling machinery. This may change in the future.
the DateTime object is defined by mxDateTime, which is a package outside the pseudo-module being "imported" by ExternalMethod.
participants (2)
-
Jim Fulton -
Wolf Logan