pickle passing over a socket
It seems to me that it could be usefull to have a zope method for passing python pickles to external processes through a unix domain socket. My thinking was to use this as a means to abstract a data base connection. Yes, I know there are zsql methods, sqlalchemy and other products to do this. But my thought was to have a generilzed method for just passing a pickle to an external process that could do anything with it. Perhaps it could pass a pickle to a queue that would handle the pickle if the site need some high volume handling of data. Then one could have another python process listening on the socket -- ready to receive pickles for zope. Does this sound like something that could be usefull -- or is it reduntant with some other feature that zope may have for sending data to external processes? -- David Bear College of Public Programs at ASU 602-464-0424
--On 22. August 2008 09:33:21 -0700 David Bear <David.Bear@asu.edu> wrote:
It seems to me that it could be usefull to have a zope method for passing python pickles to external processes through a unix domain socket. My thinking was to use this as a means to abstract a data base connection. Yes, I know there are zsql methods, sqlalchemy and other products to do this. But my thought was to have a generilzed method for just passing a pickle to an external process that could do anything with it. Perhaps it could pass a pickle to a queue that would handle the pickle if the site need some high volume handling of data.
Ever heard of XMLRPC? Sorry, your idea sounds like a bit like broken-by-design or like I-have-never-heard--anything-of-higher-protocols. -aj
On 8/22/08, Andreas Jung <lists@zopyx.com> wrote:
--On 22. August 2008 09:33:21 -0700 David Bear <David.Bear@asu.edu> wrote:
It seems to me that it could be usefull to have a zope method for passing
python pickles to external processes through a unix domain socket. My thinking was to use this as a means to abstract a data base connection. Yes, I know there are zsql methods, sqlalchemy and other products to do this. But my thought was to have a generilzed method for just passing a pickle to an external process that could do anything with it. Perhaps it could pass a pickle to a queue that would handle the pickle if the site need some high volume handling of data.
Ever heard of XMLRPC? Sorry, your idea sounds like a bit like broken-by-design or like I-have-never-heard--anything-of-higher-protocols.
You are correct, there are higher level protocols. However, since I was only thinking of using python processes, I thought I could simplify the whole design by just passing pickles. No need to parse an xml object, use fast cpickle module. -aj
-- David Bear College of Public Programs at ASU 602-464-0424
--On 22. August 2008 10:35:18 -0700 David Bear <David.Bear@asu.edu> wrote:
You are correct, there are higher level protocols. However, since I was only thinking of using python processes, I thought I could simplify the whole design by just passing pickles. No need to parse an xml object, use fast cpickle module.
I doubt that you have real reasons for reinventing the wheel. Look at Jankos posting and take one of the existing solutions. They are likely more suitable, more functional and better suited that any of your hand-crafted code. -aj
+-------[ David Bear ]---------------------- | It seems to me that it could be usefull to have a zope method for passing | python pickles to external processes through a unix domain socket. My thinking | was to use this as a means to abstract a data base connection. Yes, I know | there are zsql methods, sqlalchemy and other products to do this. But my | thought was to have a generilzed method for just passing a pickle to an | external process that could do anything with it. Perhaps it could pass a pickle | to a queue that would handle the pickle if the site need some high volume | handling of data. | | Then one could have another python process listening on the socket -- ready to | receive pickles for zope. Isn't this called ZEO? d8) -- Andrew Milton akm@theinternet.com.au
On 8/22/08, Andrew Milton <akm@theinternet.com.au> wrote:
+-------[ David Bear ]----------------------
| It seems to me that it could be usefull to have a zope method for passing | python pickles to external processes through a unix domain socket. My thinking | was to use this as a means to abstract a data base connection. Yes, I know | there are zsql methods, sqlalchemy and other products to do this. But my | thought was to have a generilzed method for just passing a pickle to an | external process that could do anything with it. Perhaps it could pass a pickle | to a queue that would handle the pickle if the site need some high volume | handling of data. | | Then one could have another python process listening on the socket -- ready to | receive pickles for zope.
Isn't this called ZEO? d8)
ZEO could be a model for this - but ZEO stores the pickles to zodb. What if I wanted to queue the pickle up and take some kind of long term processing action on it? Store it in a network file system -- which may have a high write latency to, put it in a data base for which there were no zope connectors, etc. -- David Bear College of Public Programs at ASU 602-464-0424
--On 22. August 2008 10:43:26 -0700 David Bear <David.Bear@asu.edu> wrote:
On 8/22/08, Andrew Milton <akm@theinternet.com.au> wrote:
+-------[ David Bear ]----------------------
| It seems to me that it could be usefull to have a zope method for | passing python pickles to external processes through a unix domain | socket. My thinking | was to use this as a means to abstract a data base connection. Yes, I know | there are zsql methods, sqlalchemy and other products to do this. But | my thought was to have a generilzed method for just passing a pickle | to an external process that could do anything with it. Perhaps it | could pass a pickle | to a queue that would handle the pickle if the site need some high | volume handling of data. | | Then one could have another python process listening on the socket -- ready to | receive pickles for zope.
Isn't this called ZEO? d8)
ZEO could be a model for this - but ZEO stores the pickles to zodb. What if I wanted to queue the pickle up and take some kind of long term processing action on it? Store it in a network file system -- which may have a high write latency to, put it in a data base for which there were no zope connectors, etc.
Please be _very_ detailed about your particular use. At least from what wrote so far: this sounds like an idea of the last century. -aj
+-------[ Andreas Jung ]---------------------- | | >ZEO could be a model for this - but ZEO stores the pickles to zodb. What | >if I wanted to queue the pickle up and take some kind of long term | >processing action on it? Store it in a network file system -- which may | >have a high write latency to, put it in a data base for which there were | >no zope connectors, etc. | | Please be _very_ detailed about your particular use. At least from what | wrote so far: this sounds like an idea of the last century. Sounds to me like someone just doesn't want to learn how to use RDBMS d8) -- Andrew Milton akm@theinternet.com.au
Am 22.08.2008 um 19:43 schrieb David Bear:
On 8/22/08, Andrew Milton <akm@theinternet.com.au> wrote: +------- [ David Bear ]----------------------
| It seems to me that it could be usefull to have a zope method for passing | python pickles to external processes through a unix domain socket. My thinking | was to use this as a means to abstract a data base connection. Yes, I know | there are zsql methods, sqlalchemy and other products to do this. But my | thought was to have a generilzed method for just passing a pickle to an | external process that could do anything with it. Perhaps it could pass a pickle | to a queue that would handle the pickle if the site need some high volume | handling of data. | | Then one could have another python process listening on the socket -- ready to | receive pickles for zope.
Isn't this called ZEO? d8)
ZEO could be a model for this - but ZEO stores the pickles to zodb. What if I wanted to queue the pickle up and take some kind of long term processing action on it? Store it in a network file system -- which may have a high write latency to, put it in a data base for which there were no zope connectors, etc.
Queuing, async long term processing, eventsystem are very different requirements. They have only a small part on the protcoll level, the more complex part is the logic at the recieving end. And it is easier, to call remote procedures or methods with local data, than to push pickles down a socket and decide what to do with them. There are a number of packages wich do that and much more at a higher level. I would look at http://pyro.sourceforge.net/ http://foolscap.lothar.com/trac http://threekong.com/ On the protcoll level, there are many modern contributions from the bigheads at google and facebook, like Protobufs, thrift, amqp (http://barryp.org/software/py-amqplib/ ) HTH, __Janko
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 David Bear wrote:
It seems to me that it could be usefull to have a zope method for passing python pickles to external processes through a unix domain socket. My thinking was to use this as a means to abstract a data base connection. Yes, I know there are zsql methods, sqlalchemy and other products to do this. But my thought was to have a generilzed method for just passing a pickle to an external process that could do anything with it. Perhaps it could pass a pickle to a queue that would handle the pickle if the site need some high volume handling of data.
Then one could have another python process listening on the socket -- ready to receive pickles for zope.
Does this sound like something that could be usefull -- or is it reduntant with some other feature that zope may have for sending data to external processes?
Anything like Pyro? http://pyro.sourceforge.net/ Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIryOn+gerLs4ltQ4RArm8AJ9Zt7G7QcBHbujPNzHoRK9hUJQSRwCgiSid s/Vc+rIVCQx0UAXJ7Tty4zc= =FFfH -----END PGP SIGNATURE-----
participants (5)
-
Andreas Jung -
Andrew Milton -
David Bear -
Janko Hauser -
Tres Seaver