How to publish Zope2 products on PyPI
During our latest discussion to put my most important Zope2 products into a public repository, I have promissed to publish them on PyPI instead. Yesterday, I started work to fulfill this promiss and carefully read the PyPI related documentation -- to find out, that it is not easy to publish Zope2 products. As the name suggests, PyPI is for Python packages. And Zope2 products rightfully are no Python packages: When a Zope instance starts, it does potentially expensive things for its products. I have tiny Zope instances with a few inexpensive products that start within a few seconds. And I have huge Zope instances with lots of products which parse huge XML files or have large message catalogs and take half to one minute to start. Definitely, I do not want that all my Zope instances use the same products installed via "setuptools" somewhere under the central "site-packages". My current ideas towards a solution: We define a package prefix for Zope2 products, e.g. "zope2.products". Zope2 products are published to PyPI as "<prefix>.<productname>". We extend the Zope2 configuration with an option "additional-products" which lists the products used by the instance that are not at a standard place -- such as those installed by "setuptools" -- Dieter
--On 22. September 2007 07:59:48 +0200 Dieter Maurer <dieter@handshake.de> wrote:
During our latest discussion to put my most important Zope2 products into a public repository, I have promissed to publish them on PyPI instead. Yesterday, I started work to fulfill this promiss and carefully read the PyPI related documentation -- to find out, that it is not easy to publish Zope2 products.
As the name suggests, PyPI is for Python packages. And Zope2 products rightfully are no Python packages:
When a Zope instance starts, it does potentially expensive things for its products.
I have tiny Zope instances with a few inexpensive products that start within a few seconds. And I have huge Zope instances with lots of products which parse huge XML files or have large message catalogs and take half to one minute to start.
Definitely, I do not want that all my Zope instances use the same products installed via "setuptools" somewhere under the central "site-packages".
I came across a Plone products GSXML that is shipped as egg and configured through a slug. See the docs at <http://plone.org/products/gsxml> Isn't that a suitable solution? -aj
Andreas Jung wrote at 2007-9-22 09:00 +0200:
--On 22. September 2007 07:59:48 +0200 Dieter Maurer <dieter@handshake.de> wrote:
During our latest discussion to put my most important Zope2 products into a public repository, I have promissed to publish them on PyPI instead. Yesterday, I started work to fulfill this promiss and carefully read the PyPI related documentation -- to find out, that it is not easy to publish Zope2 products.
As the name suggests, PyPI is for Python packages. And Zope2 products rightfully are no Python packages: .... My current ideas towards a solution:
We define a package prefix for Zope2 products, e.g. "zope2.products". Zope2 products are published to PyPI as "<prefix>.<productname>".
We extend the Zope2 configuration with an option "additional-products" which lists the products used by the instance that are not at a standard place -- such as those installed by "setuptools"
I came across a Plone products GSXML that is shipped as egg and configured through a slug. See the docs at
<http://plone.org/products/gsxml>
Isn't that a suitable solution?
What does "configured through a slug" mean? Looking at the page referenced above, I could not recognize how "gsxml" is made available as a Zope2 product. Looking into the egg, it looks like a standard package and not like a Zope2 product. Let me clarify my wish: I think we need an instance specific way to tell Zope 2 that it should treat something installed via "setuptools" as an additional (Zope2) product (without treating also other things laying around there as products as well); especially during startup look for an "initalize" function and call it -- beside potential Five related magic performed for Zope 2 products (such as looking for "*.zcml" files). The "zope2.product" prefix proposed above is not essential. It only tells people looking into "site-packages" that the content of this package is supposed to be used as Zope 2 products and not as standard Python packages. -- Dieter
Am 22.09.2007 um 07:59 schrieb Dieter Maurer:
My current ideas towards a solution:
We define a package prefix for Zope2 products, e.g. "zope2.products". Zope2 products are published to PyPI as "<prefix>.<productname>".
Searching for "Products" at PyPi results in a longer list. So I think a namespace is already present.
We extend the Zope2 configuration with an option "additional- products" which lists the products used by the instance that are not at a standard place -- such as those installed by "setuptools"
There is a directive in zope.conf to add additional directories to the products namespace. To add downloaded eggs to an instance one can set them in the PYTHONPATH environment variable. Buildout does it in the "./bin/ ctrlsrcipts" These are only hints, because I also have severe difficulties to put all these things together. One point I do not understand is, how to build eggs from src checkouts in one buildout for zope2 without downloading all dependencies as eggs (which are also part of the main zope install) Is there actually a list, where buildout questions can be asked? With regards, __Janko -- Janko Hauser email: jhauser@zscout.de mobile: +49 1721 641552
--On 22. September 2007 13:14:45 +0200 Janko Hauser <jh@zscout.de> wrote:
Am 22.09.2007 um 07:59 schrieb Dieter Maurer:
My current ideas towards a solution:
We define a package prefix for Zope2 products, e.g. "zope2.products". Zope2 products are published to PyPI as "<prefix>.<productname>".
Searching for "Products" at PyPi results in a longer list. So I think a namespace is already present.
We extend the Zope2 configuration with an option "additional- products" which lists the products used by the instance that are not at a standard place -- such as those installed by "setuptools"
There is a directive in zope.conf to add additional directories to the products namespace.
To add downloaded eggs to an instance one can set them in the PYTHONPATH environment variable. Buildout does it in the "./bin/ctrlsrcipts"
These are only hints, because I also have severe difficulties to put all these things together. One point I do not understand is, how to build eggs from src checkouts in one buildout for zope2 without downloading all dependencies as eggs (which are also part of the main zope install)
I think there are currently several approaches doing products-as-eggs in the Zope 2 world - I also lost track a bit and have no idea how to do it the right way[tm].
Is there actually a list, where buildout questions can be asked?
There are discussions from time to time on the distutils-sig list. -aj
Andreas Jung wrote:
I think there are currently several approaches doing products-as-eggs in the Zope 2 world - I also lost track a bit and have no idea how to do it the right way[tm].
To my knowledge, there are only two, complementary approaches: 1. Keep the Products.* namespace - distribute your egg with a namespace package 'Products' - only works (reliably) on zope trunk/2.11+ - ZCML processing and initialize() is implicit/automated 2. Use a regular python package (whether with a namespace or not) - distribute your egg with any namespace (or no namespace) - most of the plone.app.* packages are products - works on Zope 2.10.4+ - ZCML is only processed if the package is explicitly included from site.zcml (or a ZCML slug, or another package/product that is being processed) - The package only becomes a product if the <five:registerPackage /> ZCML directive is in use Martin -- Acquisition is a jealous mistress
--On 22. September 2007 12:40:04 +0100 Martin Aspeli <optilude@gmx.net> wrote:
Andreas Jung wrote:
I think there are currently several approaches doing products-as-eggs in the Zope 2 world - I also lost track a bit and have no idea how to do it the right way[tm].
To my knowledge, there are only two, complementary approaches:
1. Keep the Products.* namespace
- distribute your egg with a namespace package 'Products' - only works (reliably) on zope trunk/2.11+ - ZCML processing and initialize() is implicit/automated
2. Use a regular python package (whether with a namespace or not)
- distribute your egg with any namespace (or no namespace) - most of the plone.app.* packages are products - works on Zope 2.10.4+ - ZCML is only processed if the package is explicitly included from site.zcml (or a ZCML slug, or another package/product that is being processed) - The package only becomes a product if the <five:registerPackage /> ZCML directive is in use
Tnx for clarification. Another question: what's the point not using entry points since service discovery is one of the reasons for using entry points. Ok, with globally installed eggs this will lead to a disaster...however for isolated installations via buildout or something similar using entry points might be useful. -aj
Andreas Jung wrote:
--On 22. September 2007 12:40:04 +0100 Martin Aspeli <optilude@gmx.net> wrote:
Andreas Jung wrote:
I think there are currently several approaches doing products-as-eggs in the Zope 2 world - I also lost track a bit and have no idea how to do it the right way[tm]. To my knowledge, there are only two, complementary approaches:
1. Keep the Products.* namespace
- distribute your egg with a namespace package 'Products' - only works (reliably) on zope trunk/2.11+ - ZCML processing and initialize() is implicit/automated
2. Use a regular python package (whether with a namespace or not)
- distribute your egg with any namespace (or no namespace) - most of the plone.app.* packages are products - works on Zope 2.10.4+ - ZCML is only processed if the package is explicitly included from site.zcml (or a ZCML slug, or another package/product that is being processed) - The package only becomes a product if the <five:registerPackage /> ZCML directive is in use
Tnx for clarification. Another question: what's the point not using entry points since service discovery is one of the reasons for using entry points. Ok, with globally installed eggs this will lead to a disaster...however for isolated installations via buildout or something similar using entry points might be useful.
I agree that this seems a natural fit. I think Philipp experimented with it, but when it turned out that Products.* scanning worked even in eggs that used Products.* as a namespace package, he decided there wasn't much point in continuing. So: Entry points could potentially do what the <five:registerPackage /> directive does to register products. However, we already have that (and the magic Products.* scanning) so there may not be that much point. Entry points could also be an alternative to ZCML slugs. The downside is that this may remove the ability to disable ZCML processing in some cases. Martin -- Acquisition is a jealous mistress
Martin Aspeli wrote:
Andreas Jung wrote:
--On 22. September 2007 12:40:04 +0100 Martin Aspeli <optilude@gmx.net> wrote:
Andreas Jung wrote:
I think there are currently several approaches doing products-as-eggs in the Zope 2 world - I also lost track a bit and have no idea how to do it the right way[tm]. To my knowledge, there are only two, complementary approaches:
1. Keep the Products.* namespace
- distribute your egg with a namespace package 'Products' - only works (reliably) on zope trunk/2.11+ - ZCML processing and initialize() is implicit/automated
2. Use a regular python package (whether with a namespace or not)
- distribute your egg with any namespace (or no namespace) - most of the plone.app.* packages are products - works on Zope 2.10.4+ - ZCML is only processed if the package is explicitly included from site.zcml (or a ZCML slug, or another package/product that is being processed) - The package only becomes a product if the <five:registerPackage /> ZCML directive is in use
Tnx for clarification. Another question: what's the point not using entry points since service discovery is one of the reasons for using entry points. Ok, with globally installed eggs this will lead to a disaster...however for isolated installations via buildout or something similar using entry points might be useful.
I agree that this seems a natural fit. I think Philipp experimented with it, but when it turned out that Products.* scanning worked even in eggs that used Products.* as a namespace package, he decided there wasn't much point in continuing.
So: Entry points could potentially do what the <five:registerPackage /> directive does to register products. However, we already have that (and the magic Products.* scanning) so there may not be that much point.
Exactly. -- http://worldcookery.com -- Professional Zope documentation and training
Dieter Maurer wrote:
During our latest discussion to put my most important Zope2 products into a public repository, I have promissed to publish them on PyPI instead. Yesterday, I started work to fulfill this promiss and carefully read the PyPI related documentation -- to find out, that it is not easy to publish Zope2 products.
As the name suggests, PyPI is for Python packages. And Zope2 products rightfully are no Python packages:
When a Zope instance starts, it does potentially expensive things for its products.
I have tiny Zope instances with a few inexpensive products that start within a few seconds. And I have huge Zope instances with lots of products which parse huge XML files or have large message catalogs and take half to one minute to start.
Definitely, I do not want that all my Zope instances use the same products installed via "setuptools" somewhere under the central "site-packages".
My current ideas towards a solution:
We define a package prefix for Zope2 products, e.g. "zope2.products".
-1 to more magic namespaces!
Zope2 products are published to PyPI as "<prefix>.<productname>".
We already have that. Products.*. Basically, on Zope 2 trunk (and in 2.11), we've made Products/__init__.py declare Products.* as a namespace package. That means that you can package up an egg that uses this namespace package and upload it to the cheese shop. Several of the Plone packages, e.g. http://cheeseshop.python.org/pypi/Products.CMFDynamicViewFTI/3.0, do this.
We extend the Zope2 configuration with an option "additional-products" which lists the products used by the instance that are not at a standard place -- such as those installed by "setuptools"
I really don't see the need for this kind of complication. I would never install a Products.* package into the global python interpreter. That's a recipe for pain and incompatibility. Instead, I'd use zc.buildout or virtualenv to have a sandbox for these eggs. see http://plone.org/documentation/tutorial/buildout and http://martinaspeli.net/articles/python-package-management. Secondly, we already have something like additional-products, in that for packages outside the Products.* namespace, we do not automatically read ZCML, call initialize() and so on. Instead, we require that: - the package's ZCML is explicitly pulled in, either via a ZCML slug, or via an include from another ZCML file which is explicitly (or implicitly, if it's in Products/*) parsed. - the package uses the <five:registerPackage /> directive to declare the package a Zope 2 product, which causes its initialize() function to be called at the appropriate time. Note that anything in Products.*, whether eggs in that namespace package or plain directories in e.g. $INSTANCE_HOME/Products, behave as they always have: ZCML is auto-loaded by Five, initialize() is always called. Martin -- Acquisition is a jealous mistress
Martin Aspeli wrote at 2007-9-22 12:21 +0100:
Dieter Maurer wrote: We extend the Zope2 configuration with an option "additional-products" which lists the products used by the instance that are not at a standard place -- such as those installed by "setuptools"
I really don't see the need for this kind of complication. I would never install a Products.* package into the global python interpreter. That's a recipe for pain and incompatibility. Instead, I'd use zc.buildout or virtualenv to have a sandbox for these eggs.
We definitely have different views on "complication". In my view "buildout" or "virtualenv" are much more complication than an additional option in "zope.conf". "Five:registerPackage" may be used as equivalent to the proposed new Zope configuration option "additional-products". Personally, I find it more natural to have the declaration in "zope.conf" -- for two reasons: * other product controlling options are there already (the "products" key) * no need to involve ZCML (which adds significant complexity) But, maybe, the wrong way has already been taken -- and we want to stick to it ;-) -- Dieter
Dieter Maurer wrote:
Martin Aspeli wrote at 2007-9-22 12:21 +0100:
Dieter Maurer wrote: We extend the Zope2 configuration with an option "additional-products" which lists the products used by the instance that are not at a standard place -- such as those installed by "setuptools" I really don't see the need for this kind of complication. I would never install a Products.* package into the global python interpreter. That's a recipe for pain and incompatibility. Instead, I'd use zc.buildout or virtualenv to have a sandbox for these eggs.
We definitely have different views on "complication". In my view "buildout" or "virtualenv" are much more complication than an additional option in "zope.conf".
*shrug* - at least they're much more general, not Zope-specific and thus not something we need to build it and maintain.
"Five:registerPackage" may be used as equivalent to the proposed new Zope configuration option "additional-products". Personally, I find it more natural to have the declaration in "zope.conf" -- for two reasons:
* other product controlling options are there already (the "products" key)
* no need to involve ZCML (which adds significant complexity)
But, maybe, the wrong way has already been taken -- and we want to stick to it ;-)
I guess it's a matter of opinion. I think you'll find that as eggs get more and more important in the Python world in general, and also in Zope, we need to leverage them rather than fight them. I've found it easier to work with (and certainly to redistribute!) eggs and simpler packages than simple products, but it is a bit of a shift, and obviously not everyone is going to be happy. However, using buildout, for example, is not very difficult, and using something like virtualenv is even simpler. It's like one extra command. :) Martin -- Acquisition is a jealous mistress
On 9/22/07, Dieter Maurer <dieter@handshake.de> wrote:
Definitely, I do not want that all my Zope instances use the same products installed via "setuptools" somewhere under the central "site-packages".
Right. Does a listing on CheeseShop automatically mean that it has to be installable with setuptools? But in any case, the packages you get, the Products.* existing there already, have a structure of: /Products.TheProduct /Products.TheProduct/Product /Products.TheProduct/Product/TheProduct For small sites not using buildout or so, downloading this tgz and moving TheProduct directory into your Products directory works and is no more work that having a tgz with a /TheProduct-4.5.6 directory, wich is quite common also. So with this you can choose if you want to use setuptoools or not...
Lennart Regebro wrote at 2007-9-22 16:02 +0200:
... Right. Does a listing on CheeseShop automatically mean that it has to be installable with setuptools?
No, but it would make things easier....
But in any case, the packages you get, the Products.* existing there already, have a structure of:
/Products.TheProduct /Products.TheProduct/Product /Products.TheProduct/Product/TheProduct
Thanks. This concludes the namespace question. In my view, something like "zope2.product" would be more explicit, but "Products.", too, may do -- users of "PyPI" can see that these things work only for Zope2 from the "Framework::Zope2" classifier. -- Dieter
--On 23. September 2007 08:24:56 +0200 Dieter Maurer <dieter@handshake.de> wrote:
In my view, something like "zope2.product" would be more explicit, but "Products.", too, may do -- users of "PyPI" can see that these things work only for Zope2 from the "Framework::Zope2" classifier.
Adding an additional classifier is perhaps the best solution. Prefixes are already used for indicating the ownership of a package. However I see a minor discrepancy between the naming of products - at least on svn.zope.org where all Zope 2 products have a top-level name "Product."<product name>. My SQLAlchemyDA product lives on svn.zope.org under Product.SQLAlchemyDA. If I should release it on PyPI I might use a name like zopyx.SQLAlchemyDA. The uppercase notation is usually used for Zope products, lowercase notation usually indicates a Python package. -aj
Dieter Maurer wrote:
During our latest discussion to put my most important Zope2 products into a public repository, I have promissed to publish them on PyPI instead. Yesterday, I started work to fulfill this promiss and carefully read the PyPI related documentation -- to find out, that it is not easy to publish Zope2 products.
As the name suggests, PyPI is for Python packages. And Zope2 products rightfully are no Python packages:
Sure they are. They're a directory with an __init__.py. I think this is a misunderstanding. It seems you think Python packages have to be installed globally. This isn't the case. And certainly nobody would recommend doing that for Products packages.
When a Zope instance starts, it does potentially expensive things for its products.
This would only be a problem if the Products packages were installed globally. They don't have to. See below.
I have tiny Zope instances with a few inexpensive products that start within a few seconds. And I have huge Zope instances with lots of products which parse huge XML files or have large message catalogs and take half to one minute to start.
Definitely, I do not want that all my Zope instances use the same products installed via "setuptools" somewhere under the central "site-packages".
Nobody expects you to. As others have pointed out * PyPI doesn't necessarily have to contain eggs. It's primarily a discovery mechanism for humans. The fact that setuptools can download packages from it is not as important as the fact that developers can find packages there. * Installing eggs doesn't necessarily mean they're installed globally into site-packages. There are a number of mechanism to create isolated sandboxes, some of which include turning a Zope 2 instance into an egg-capable sandbox (in other words, so that lib/python can contain eggs). And then there's zc.buildout, of course.
My current ideas towards a solution:
We define a package prefix for Zope2 products, e.g. "zope2.products". Zope2 products are published to PyPI as "<prefix>.<productname>".
-1 +1 for *continuing* to use the already widely used Products prefix.
We extend the Zope2 configuration with an option "additional-products" which lists the products used by the instance that are not at a standard place -- such as those installed by "setuptools"
As long as the packages are on the PYTHONPATH, Zope 2 will find them. I don't think there's a need for this new directive. -- http://worldcookery.com -- Professional Zope documentation and training
Philipp von Weitershausen wrote at 2007-9-22 19:27 +0200:
... Dieter Maurer wrote: ... * PyPI doesn't necessarily have to contain eggs. It's primarily a discovery mechanism for humans. The fact that setuptools can download packages from it is not as important as the fact that developers can find packages there.
The reason why I had to promiss to make my products available via PyPI was that they can be downloaded from there.
...
We extend the Zope2 configuration with an option "additional-products" which lists the products used by the instance that are not at a standard place -- such as those installed by "setuptools"
As long as the packages are on the PYTHONPATH, Zope 2 will find them. I don't think there's a need for this new directive.
But, such packages are not treated as Zope2 products -- which is essential for packages that are to be used as Zope2 products. I have learned meanwhile that by some disregard of concerns, Five can turn a package into a Zope2 product. Thus, the functionality is there -- just maybe in the wrong place. -- Dieter
On 23 Sep 2007, at 19:05 , Dieter Maurer wrote:
Philipp von Weitershausen wrote at 2007-9-22 19:27 +0200:
... Dieter Maurer wrote: ... * PyPI doesn't necessarily have to contain eggs. It's primarily a discovery mechanism for humans. The fact that setuptools can download packages from it is not as important as the fact that developers can find packages there.
The reason why I had to promiss to make my products available via PyPI was that they can be downloaded from there.
Right. So then I don't understand why need all that extra machinery in Zope and an extra namespace.
We extend the Zope2 configuration with an option "additional- products" which lists the products used by the instance that are not at a standard place -- such as those installed by "setuptools"
As long as the packages are on the PYTHONPATH, Zope 2 will find them. I don't think there's a need for this new directive.
But, such packages are not treated as Zope2 products -- which is essential for packages that are to be used as Zope2 products.
If a package is in the Products.* namespace, Zope 2 will find it and load it. No matter where on the PYTHONPATH it is located. So I think my statement is correct.
I have learned meanwhile that by some disregard of concerns, Five can turn a package into a Zope2 product.
Yup. You can put your software into an arbitrarily named Python package (e.g. dieter.mystuff) and have it be treated like a Zope 2 product (which really just means calling an initialize() function at some point).
Thus, the functionality is there -- just maybe in the wrong place.
If Five's registerPackage functionality is what you had been proposing all along, then I have been misunderstanding you terribly. Either way, you don't elaborate on why you think that this is the "wrong" place for it. I personally consider registerPackage a solutino for integrating new software (Zope 3-style software in Python packages) into a legacy discovery mechanism (the automatic Products.* loading).
Philipp von Weitershausen wrote at 2007-9-23 19:24 +0200:
On 23 Sep 2007, at 19:05 , Dieter Maurer wrote:
Philipp von Weitershausen wrote at 2007-9-22 19:27 +0200:
... Dieter Maurer wrote: ... * PyPI doesn't necessarily have to contain eggs. It's primarily a discovery mechanism for humans. The fact that setuptools can download packages from it is not as important as the fact that developers can find packages there.
The reason why I had to promiss to make my products available via PyPI was that they can be downloaded from there.
Right. So then I don't understand why need all that extra machinery in Zope and an extra namespace.
Others already have pointed out, that the namespace already exists in "PyPI": "Products". It was not obvious ;-) I am fine with this. "all that extra machinery" means a declaration "additional-product[s]" -- something I have learned meanwhile exists already in Five as "registerProduct". I explain below why I think "Five" is not the proper place.
....
Thus, the functionality is there -- just maybe in the wrong place.
If Five's registerPackage functionality is what you had been proposing all along, then I have been misunderstanding you terribly. Either way, you don't elaborate on why you think that this is the "wrong" place for it. I personally consider registerPackage a solutino for integrating new software (Zope 3-style software in Python packages) into a legacy discovery mechanism (the automatic Products.* loading).
I have already sketched my reason why I deem "Five" the wrong place to a "registerProduct". I repeat it again: * Five is the vehicle to make Zope 3 concepts available in Zope 2. A product is a Zope 2 concept. Zope 3 does not have it. Thus why, what has Five to do with Zope 2 products? * "zope.conf" has already a declaration that controls which products are used by an instance "products". When we need another one (I do feel this need), I would have put it there, in "zope.conf", where already the other declaration lives -- and not in "Five". Having one declaration in "zope.conf" and one in "Five" looks like bad design for me... -- Dieter
participants (6)
-
Andreas Jung -
Dieter Maurer -
Janko Hauser -
Lennart Regebro -
Martin Aspeli -
Philipp von Weitershausen