A suggestion for changing exception handler when loading 'c' optimisations to make modules more friendly for GAE
Hi all Can I make a suggestion that would make a couple of modules more gae friendly ;-) zope.interface and zope.i18nmessageid have 'c' optimisations which obviously don't work under app engine. When these modules are imported you get the following exception. gae/1.333250465889549129/zope/i18nmessageid/_zope_i18nmessageid_message.py", line 6, in __bootstrap__ imp.load_dynamic(__name__,__file__) File "/base/python_dist/lib/python2.5/py_imp.py", line 116, in load_dynamic raise NotImplementedError('This function is not supported on App Engine.') Unfortunately the code (in this case in zope.i18nmessageid.Message) does the following which doesn't deal with the Exception raised by the app engine runtime. try: from _zope_i18nmessageid_message import Message except ImportError: pass To make it more "gae friendly" I think we should change these occurrences to try: from _zope_i18nmessageid_message import Message except ImportError, NotImplementedError: pass Regards Tim
Hi, yes, I'd really appreciate such a modification. Then I could delete the regarding code in my rod.recipe.appengine, which provides a dummy implementation of imp.load_dynamic. Tim, do you have a list of the concerning packages? I could help making these changes. A good reason to file an application for commit rights :-) Regards Tobias On 03.07.2009, at 02:39, Tim Hoffman wrote:
Hi all
Can I make a suggestion that would make a couple of modules more gae friendly ;-) zope.interface and zope.i18nmessageid have 'c' optimisations which obviously don't work under app engine.
When these modules are imported you get the following exception.
gae/1.333250465889549129/zope/i18nmessageid/ _zope_i18nmessageid_message.py", line 6, in __bootstrap__ imp.load_dynamic(__name__,__file__) File "/base/python_dist/lib/python2.5/py_imp.py", line 116, in load_dynamic raise NotImplementedError('This function is not supported on App Engine.')
Unfortunately the code (in this case in zope.i18nmessageid.Message) does the following which doesn't deal with the Exception raised by the app engine runtime.
try: from _zope_i18nmessageid_message import Message except ImportError: pass
To make it more "gae friendly" I think we should change these occurrences to
try: from _zope_i18nmessageid_message import Message except ImportError, NotImplementedError: pass
Regards
Tim _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Tim Hoffman wrote:
gae/1.333250465889549129/zope/i18nmessageid/_zope_i18nmessageid_message.py", line 6, in __bootstrap__ imp.load_dynamic(__name__,__file__) File "/base/python_dist/lib/python2.5/py_imp.py", line 116, in load_dynamic raise NotImplementedError('This function is not supported on App Engine.')
This actually feels like a bug in GAE to me... This should be an ImportError... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
I suppose that depends on your point of view. They exception is actually saying that imp.load_dynamic is not implemented, not that it can't load the dynamic module so I would say in my opinion that it isn't a bug. Rgds Tim On Fri, Jul 3, 2009 at 4:38 PM, Chris Withers<chris@simplistix.co.uk> wrote:
Tim Hoffman wrote:
gae/1.333250465889549129/zope/i18nmessageid/_zope_i18nmessageid_message.py", line 6, in __bootstrap__ imp.load_dynamic(__name__,__file__) File "/base/python_dist/lib/python2.5/py_imp.py", line 116, in load_dynamic raise NotImplementedError('This function is not supported on App Engine.')
This actually feels like a bug in GAE to me... This should be an ImportError...
Chris
-- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
On 03.07.2009, at 10:38, Chris Withers wrote:
Tim Hoffman wrote:
gae/1.333250465889549129/zope/i18nmessageid/ _zope_i18nmessageid_message.py", line 6, in __bootstrap__ imp.load_dynamic(__name__,__file__) File "/base/python_dist/lib/python2.5/py_imp.py", line 116, in load_dynamic raise NotImplementedError('This function is not supported on App Engine.')
This actually feels like a bug in GAE to me... This should be an ImportError...
Sorry, I don't agree. We assume an imp.load_dynamic function from what we read in the python documentation. But the implementation in GAE does not provide the expected api. So the NotImplementedError seems absolutely correct to me (see http://www.python.org/doc/2.5.2/lib/module-exceptions.html#l2h-114) . Tobias
Tobias Rodäbel wrote:
Sorry, I don't agree. We assume an imp.load_dynamic function from what we read in the python documentation. But the implementation in GAE does not provide the expected api. So the NotImplementedError seems absolutely correct to me (see http://www.python.org/doc/2.5.2/lib/module-exceptions.html#l2h-114)
Ahhh, I see, sorry, misread where the exception was coming from. Surely the try:except for the NotImplementedError should be rwapped just around the call to imp.load_dynamic, then? ie: try: imp.load_dynamic(__name__,__file__) except NotImplementedError: raise ImportError('load_dynamic not available so C extension cannot be used') cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On Jul 2, 2009, at 8:39 PM, Tim Hoffman wrote:
Hi all
Can I make a suggestion that would make a couple of modules more gae friendly ;-) zope.interface and zope.i18nmessageid have 'c' optimisations which obviously don't work under app engine.
When these modules are imported you get the following exception.
gae/1.333250465889549129/zope/i18nmessageid/ _zope_i18nmessageid_message.py", line 6, in __bootstrap__ imp.load_dynamic(__name__,__file__) File "/base/python_dist/lib/python2.5/py_imp.py", line 116, in load_dynamic raise NotImplementedError('This function is not supported on App Engine.')
This bootstrapping code is generated by setuptools to deal with loading extension modules in zip files. It does some unholy things to make this work. I'd really like to find a way to suppress this. In general, I find deploying as zip files to be an anti-feature. In seems especially insane when extension modules are involved.
Unfortunately the code (in this case in zope.i18nmessageid.Message) does the following which doesn't deal with the Exception raised by the app engine runtime.
You mean with the exception raised by the setuptools wrapper code.
try: from _zope_i18nmessageid_message import Message except ImportError: pass
To make it more "gae friendly" I think we should change these occurrences to
try: from _zope_i18nmessageid_message import Message except ImportError, NotImplementedError: pass
This might be an ok work around. How hard would it be to just remove the wrapper script when deploying to app engine? Jim -- Jim Fulton Zope Corporation
Hi Jim That is what I am doing now, the problem I see though is when someone upates (via buildout etc.... a) module they need to remember to remove the recreated file before redeploying the app, unfortunately I am not sure I see a way reliable way of specifying that these files should be deleted in any automatic sense. At least an ignore in the app.yaml isn't feasible. Tim ...
This might be an ok work around.
How hard would it be to just remove the wrapper script when deploying to app engine?
Jim
-- Jim Fulton Zope Corporation
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
On Jul 3, 2009, at 6:25 AM, Tim Hoffman wrote:
Hi Jim
That is what I am doing now, the problem I see though is when someone upates (via buildout etc.... a) module they need to remember to remove the recreated file before redeploying the app, unfortunately I am not sure I see a way reliable way of specifying that these files should be deleted in any automatic sense. At least an ignore in the app.yaml isn't feasible.
I'm sure there's a way to do it. It will just take some digging. Maybe there's a way to tell setuptools not to do it or maybe there's enough meta data laying around for buildout to undo it. For example, the EGG-INFO/SOURCES.txt has a list of the original files in the distribution, so it should be easy enough to figure out where these extra bogus files are. Jim -- Jim Fulton Zope Corporation
On 03.07.2009, at 12:33, Jim Fulton wrote:
On Jul 3, 2009, at 6:25 AM, Tim Hoffman wrote:
Hi Jim
That is what I am doing now, the problem I see though is when someone upates (via buildout etc.... a) module they need to remember to remove the recreated file before redeploying the app, unfortunately I am not sure I see a way reliable way of specifying that these files should be deleted in any automatic sense. At least an ignore in the app.yaml isn't feasible.
I'm sure there's a way to do it. It will just take some digging. Maybe there's a way to tell setuptools not to do it or maybe there's enough meta data laying around for buildout to undo it. For example, the EGG-INFO/SOURCES.txt has a list of the original files in the distribution, so it should be easy enough to figure out where these extra bogus files are.
Great idea to examine EGG-INFO/SOURCES.txt, Jim! I just added this to my recipe (http://pypi.python.org/pypi/rod.recipe.appengine) and got rid of the horrible monkey patch for imp.load_dynamic. Now it does exactly what Tim wants and excludes the optional c extension stuff (plus .pyo, and .pyc files) in a gae buildout. I'm planning to add an option for not deploying the additional packes as a zip archive if the gae file limit will not be exceeded. Cheers, Tobias
Hi Tobias Cool Just something to note google recently upped the file limit to 3000 so don't be too agressive generating those zips ;-) T
Great idea to examine EGG-INFO/SOURCES.txt, Jim! I just added this to my recipe (http://pypi.python.org/pypi/rod.recipe.appengine) and got rid of the horrible monkey patch for imp.load_dynamic. Now it does exactly what Tim wants and excludes the optional c extension stuff (plus .pyo, and .pyc files) in a gae buildout. I'm planning to add an option for not deploying the additional packes as a zip archive if the gae file limit will not be exceeded.
Cheers, Tobias
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jim Fulton wrote:
On Jul 2, 2009, at 8:39 PM, Tim Hoffman wrote:
Hi all
Can I make a suggestion that would make a couple of modules more gae friendly ;-) zope.interface and zope.i18nmessageid have 'c' optimisations which obviously don't work under app engine.
When these modules are imported you get the following exception.
gae/1.333250465889549129/zope/i18nmessageid/ _zope_i18nmessageid_message.py", line 6, in __bootstrap__ imp.load_dynamic(__name__,__file__) File "/base/python_dist/lib/python2.5/py_imp.py", line 116, in load_dynamic raise NotImplementedError('This function is not supported on App Engine.')
I think the suggestion to remap the NotImplementedError to an Import error is sensible (in that __bootstrap__ method).
This bootstrapping code is generated by setuptools to deal with loading extension modules in zip files. It does some unholy things to make this work. I'd really like to find a way to suppress this.
In general, I find deploying as zip files to be an anti-feature. In seems especially insane when extension modules are involved.
The only way to deploy *any* non-toy app on GAE is via zip file(s) (I would think a single big one would be best), due to their limit on the number of files. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKTkAH+gerLs4ltQ4RAhazAJ4itN+wO1rPwXPyLIEIaNF6VKSScgCdGyfd OxfF8AUnMYT09/L2B5RF9O0= =RGfd -----END PGP SIGNATURE-----
On 7/3/09 1:29 PM, Tres Seaver wrote:
In general, I find deploying as zip files to be an anti-feature. In seems especially insane when extension modules are involved.
The only way to deploy *any* non-toy app on GAE is via zip file(s) (I would think a single big one would be best), due to their limit on the number of files.
Actually they've raised the limit to 3000 files now, which is more realistic (previously it was impossible to deploy even minimal apps without zipping library eggs, now the zipping is largely unnecessary if you keep your dependencies sane). - C
participants (6)
-
Chris McDonough -
Chris Withers -
Jim Fulton -
Tim Hoffman -
Tobias Rodäbel -
Tres Seaver