Proposal: merge zc.configuration's exclude directive into zope.configuration.
Hi there. The "exclude" directive provided by zc.configuration package is easy to use and straightforward. I think it's used almost in every zope-based application setup. Its implementation is very small and fits great in zope.configuration's standard directives. So I'd like to propose to move it to zope.configuration and make it always available for ZMCL files, just like the "include" directive, so people would'nt need to install extra package and include an extra "meta.zcml" file before being able to use it. If noone objects, I'd like to do that and make the "exclude" directive from zc.configuration point to the zope.configuration's implementation making the original place deprecated (however I guess the whole zc.configuration package should't be deprecated as it's intended to be a common place for configuration extensions, even if it has only one directive now). Jim, if you're fine with that, can you please give me rights for zc.configuration on PYPI, my user name is "nadako". -- WBR, Dan Korostelev
Hey, Dan Korostelev wrote:
The "exclude" directive provided by zc.configuration package is easy to use and straightforward. I think it's used almost in every zope-based application setup.
I highly doubt so; I don't find myself using it a lot myself, for instance. :)
Its implementation is very small and fits great in zope.configuration's standard directives. So I'd like to propose to move it to zope.configuration and make it always available for ZMCL files, just like the "include" directive, so people would'nt need to install extra package and include an extra "meta.zcml" file before being able to use it.
+1 (note though that including an extra "meta.zcml" can be avoided if you make use of the z3c.autoinclude library) Regards, Martijn
2009/2/26 Martijn Faassen <faassen@startifact.com>:
Dan Korostelev wrote:
The "exclude" directive provided by zc.configuration package is easy to use and straightforward. I think it's used almost in every zope-based application setup.
I highly doubt so; I don't find myself using it a lot myself, for instance. :)
Well, may be, really not in "every app", but it's still used alot. :-)
(note though that including an extra "meta.zcml" can be avoided if you make use of the z3c.autoinclude library)
Yep, I know about z3c.autoinclude, but I don't like it, as it makes things more implicit and it also slows down startup time for applications that uses many eggs. I'd like to see an option for packages that are using z3c.autoinclude to make autoincluding conditional, so those who doesn't like it or needs more control could just turn it off. -- WBR, Dan Korostelev
Hey, On Thu, Feb 26, 2009 at 12:43 PM, Dan Korostelev <nadako@gmail.com> wrote: [snip]
(note though that including an extra "meta.zcml" can be avoided if you make use of the z3c.autoinclude library)
Yep, I know about z3c.autoinclude, but I don't like it, as it makes things more implicit and it also
Yes, automation makes things more implicit. This is *not* an argument that can be used against any and all automation. An explicit includeDependencies directive will include the configure.zcml and meta.zcml of packages that are dependencies of that package. You make less mistakes this way (and it's very common to make the mistake to forget inclusion of some ZCML). It's also pretty rare to want to do otherwise in my experience (and there's always the exclude directive). I'd also argue auto-inclusion can actually help guard against dependency bugs, such as cases where a package tries to include ZCML it doesn't depend on (and works because some other package makes the dependency installed).
slows down startup time for applications that uses many eggs.
When claims like that are made, I'd like to see measurements that demonstrate significant slowdowns during startup. Undoubtedly more code is excuted than when you write out 'include' directives manually, but is the slowdown actually measurable?
I'd like to see an option for packages that are using z3c.autoinclude to make autoincluding conditional, so those who doesn't like it or needs more control could just turn it off.
That's asking for a feature that other packages that *don't* use autoinclude don't support! You lose control as soon as you include a package's "configure.zcml". There's no difference with non-auto-inclusion here; if you include a package's configure.zcml you get whatever it includes, automatically or using normal zcml include directives. When auto-inclusion is used all dependencies that use setup.py are included too. If you want to change that behavior, you will have to skip importing that package's configure.zcml altogether, just like in the case where that package does write its include statements out explicitly. Anyway, it's fine if you don't want to use auto-inclusion in a package, but it's not as arbitrarily magic as you are suggesting here. Regards, Martijn
On Thu, Feb 26, 2009 at 7:07 AM, Martijn Faassen <faassen@startifact.com> wrote:
Hey,
On Thu, Feb 26, 2009 at 12:43 PM, Dan Korostelev <nadako@gmail.com> wrote:
Yep, I know about z3c.autoinclude, but I don't like it, as it makes things more implicit and it also
Yes, automation makes things more implicit. This is *not* an argument that can be used against any and all automation. An explicit includeDependencies directive will include the configure.zcml and meta.zcml of packages that are dependencies of that package. You make less mistakes this way (and it's very common to make the mistake to forget inclusion of some ZCML). It's also pretty rare to want to do otherwise in my experience (and there's always the exclude directive).
I'd also argue auto-inclusion can actually help guard against dependency bugs, such as cases where a package tries to include ZCML it doesn't depend on (and works because some other package makes the dependency installed).
+1 on Martijn's points (unsurprisingly, as I'm one of his co-authors on autoinclude ;) I'll also add that z3c.autoinclude tries very hard to log its actions in an easily traceable way, and I've been trying to build in more debugging tools as well.
slows down startup time for applications that uses many eggs.
When claims like that are made, I'd like to see measurements that demonstrate significant slowdowns during startup. Undoubtedly more code is excuted than when you write out 'include' directives manually, but is the slowdown actually measurable?
I've heard this claim before and I'd also love to see some measurements. I haven't really had the time to do any myself, and in my experience z3c.autoinclude hasn't slowed things down enough to bother me. But I may have an unusually high tolerance for slow-moving code. :) The slowest code in z3c.autoinclude is well isolated and is tested for many edge cases, and I'm pretty sure there are obvious optimizations that can be made without even rewriting it. So improving the execution speed should be quite easy and safe; it just hasn't been a priority for me, and I'd prefer to get some measurements before trying to tweak it.
I'd like to see an option for packages that are using z3c.autoinclude to make autoincluding conditional, so those who doesn't like it or needs more control could just turn it off.
That's asking for a feature that other packages that *don't* use autoinclude don't support! You lose control as soon as you include a package's "configure.zcml". There's no difference with non-auto-inclusion here; [snip]
That's true on paper, but in practice z3c.autoinclude's *indirection* does make a difference when you're just trying to debug what's going on. At any rate, the next release of z3c.autoinclude will provide this feature, through environment variables: http://lists.plone.org/pipermail/framework-team/2009-February/002690.html Regards, Ethan
Hey, On Thu, Feb 26, 2009 at 5:32 PM, Ethan Jucovy <ethan.jucovy@gmail.com> wrote:
On Thu, Feb 26, 2009 at 7:07 AM, Martijn Faassen <faassen@startifact.com> wrote: [snip]
That's asking for a feature that other packages that *don't* use autoinclude don't support! You lose control as soon as you include a package's "configure.zcml". There's no difference with non-auto-inclusion here; [snip]
That's true on paper, but in practice z3c.autoinclude's *indirection* does make a difference when you're just trying to debug what's going on.
If something goes wrong, that's true. I'm not sure how it goes wrong - if it goes right, you're going to have included the list of dependencies that's in setup.py (if they have relevant zcml files).
At any rate, the next release of z3c.autoinclude will provide this feature, through environment variables: http://lists.plone.org/pipermail/framework-team/2009-February/002690.html
Yeah, I saw some code. I don't quite see the point of that feature, as it will most likely result in non-working code, right? Regards, Martijn
On Fri, Feb 27, 2009 at 10:35 AM, Martijn Faassen <faassen@startifact.com> wrote:
Hey,
On Thu, Feb 26, 2009 at 5:32 PM, Ethan Jucovy <ethan.jucovy@gmail.com> wrote:
That's true on paper, but in practice z3c.autoinclude's *indirection* does make a difference when you're just trying to debug what's going on.
If something goes wrong, that's true. I'm not sure how it goes wrong - if it goes right, you're going to have included the list of dependencies that's in setup.py (if they have relevant zcml files)
At any rate, the next release of z3c.autoinclude will provide this feature, through environment variables: http://lists.plone.org/pipermail/framework-team/2009-February/002690.html
Yeah, I saw some code. I don't quite see the point of that feature, as it will most likely result in non-working code, right?
Yeah, most likely. :) I'll admit I mainly added it for symmetry with autoincludePlugins, where the case for that feature is a lot stronger, because of the control inversion that happens there. A few months ago Vincent Fretin pointed out (http://mail.zope.org/pipermail/grok-dev/2008-November/006690.html) a possible case for at least selective disabling of dependency autoinclusion, though, and I think the feature is harmless enough that it may as well exist. The only times I've used it so far have been just for fun though -- turning off autoinclusion somewhere to see what breaks. :) Ethan
Hey, On Sat, Feb 28, 2009 at 5:02 PM, Ethan Jucovy <ethan.jucovy@gmail.com> wrote: [snip]
The only times I've used it so far have been just for fun though -- turning off autoinclusion somewhere to see what breaks. :)
Dan just came up with a good use case for the feature, so now I'm glad you added it. :) It's useful to be able to turn it off and write all necessary includes manually somewhere, so that you can compare the performance in a large system. I'd really appreciate it if you could work with Dan to get performance data on autoinclude. His problems might indicate we have to introduce a bit of caching somewhere. Regards, Martijn
2009/2/26 Martijn Faassen <faassen@startifact.com>:
Hey,
On Thu, Feb 26, 2009 at 12:43 PM, Dan Korostelev <nadako@gmail.com> wrote: [snip]
(note though that including an extra "meta.zcml" can be avoided if you make use of the z3c.autoinclude library)
Yep, I know about z3c.autoinclude, but I don't like it, as it makes things more implicit and it also
Yes, automation makes things more implicit. This is *not* an argument that can be used against any and all automation.
Hey, I didn't try to argue about anything. It was just my subjective opinion. :-)
An explicit includeDependencies directive will include the configure.zcml and meta.zcml of packages that are dependencies of that package. You make less mistakes this way (and it's very common to make the mistake to forget inclusion of some ZCML). It's also pretty rare to want to do otherwise in my experience (and there's always the exclude directive).
Well, I guess that it's my personal desire of control. I'll probably change my mind about that, as (thinking now), there are really less things that I don't want to include than things that I want to include. Also, the plugin mechanism looks interesting to me now. :)
I'd also argue auto-inclusion can actually help guard against dependency bugs, such as cases where a package tries to include ZCML it doesn't depend on (and works because some other package makes the dependency installed).
Good point.
slows down startup time for applications that uses many eggs.
When claims like that are made, I'd like to see measurements that demonstrate significant slowdowns during startup. Undoubtedly more code is excuted than when you write out 'include' directives manually, but is the slowdown actually measurable?
Well, that's probably a really big mis-use (I even hope so), but I ran into really gross slowdown while working on the z3ext project. They have about 130 eggs (not counting zope 3.4 ones) and each one has the autoinclude directive and a bunch of egg dependencies. So the application startup time on my (not so fast though) computer is about 4-5 mins :-/ So, is z3c.autoinclude intended to be used by any egg, including "feature packages" (like z3c.form) or only for actual applications (like a ZMI instance)? I'd really like if this topic will be cleared up in z3c.autoinclude docs. Also, is there any caching for already processed packages in the include finder code? If no, I'd probably like to contribute some, if I'll use z3c.autoinclude. :)
I'd like to see an option for packages that are using z3c.autoinclude to make autoincluding conditional, so those who doesn't like it or needs more control could just turn it off.
That's asking for a feature that other packages that *don't* use autoinclude don't support! You lose control as soon as you include a package's "configure.zcml". There's no difference with non-auto-inclusion here; if you include a package's configure.zcml you get whatever it includes, automatically or using normal zcml include directives. When auto-inclusion is used all dependencies that use setup.py are included too. If you want to change that behavior, you will have to skip importing that package's configure.zcml altogether, just like in the case where that package does write its include statements out explicitly.
Well, it's quite useful in the "z3ext" case I described above - I could just turn off the autoincluding and carefully write the includes in the instance zcml, so get not so crazy startup time.
Anyway, it's fine if you don't want to use auto-inclusion in a package, but it's not as arbitrarily magic as you are suggesting here.
Thanks, I'll take a look at it again. -- WBR, Dan Korostelev
Hi there, On Sat, Feb 28, 2009 at 3:14 PM, Dan Korostelev <nadako@gmail.com> wrote:
2009/2/26 Martijn Faassen <faassen@startifact.com>: [snip]
When claims like that are made, I'd like to see measurements that demonstrate significant slowdowns during startup. Undoubtedly more code is excuted than when you write out 'include' directives manually, but is the slowdown actually measurable?
Well, that's probably a really big mis-use (I even hope so), but I ran into really gross slowdown while working on the z3ext project. They have about 130 eggs (not counting zope 3.4 ones) and each one has the autoinclude directive and a bunch of egg dependencies. So the application startup time on my (not so fast though) computer is about 4-5 mins :-/
Okay, that's a good data point. I haven't used autoinclude on that scale yet, but I think it *should* be usable on that scale. How many minutes got added to the startup time when autoinclude was introduced, any idea? Some algorithm in the middle of z3c.autoinclude might be slowing things down, but hopefully a caching approach could help eliminate this. We need to identify the pain point. Could you provide some form of profiling data?
So, is z3c.autoinclude intended to be used by any egg, including "feature packages" (like z3c.form) or only for actual applications (like a ZMI instance)? I'd really like if this topic will be cleared up in z3c.autoinclude docs.
I think it should be useful for any package that has ZCML dependencies at all, so if that slows things down we should fix it.
Also, is there any caching for already processed packages in the include finder code? If no, I'd probably like to contribute some, if I'll use z3c.autoinclude. :)
Ah, you're thinking in the same direction. I don't think there's any caching at all yet in autoinclude and that'd be the first thing I'd look at if I were to look into speeding it up. It's just I hadn't run into the pain point yet.
I'd like to see an option for packages that are using z3c.autoinclude to make autoincluding conditional, so those who doesn't like it or needs more control could just turn it off. [I ask "why?" not seeing the point] Well, it's quite useful in the "z3ext" case I described above - I could just turn off the autoincluding and carefully write the includes in the instance zcml, so get not so crazy startup time.
Aah, yes, I see that example. The autoinclude trunk, as Ethan said in another reply, does have support for this with an environment variable. I didn't see the point for it but now I do. :) Anyway, I hadn't realized you had more experience with autoinclude than I thought, I'm sorry for my tone. I hope we can learn from your feedback to improve the pain points you discovered. Regards, Martijn
On Mon, Mar 2, 2009 at 9:59 AM, Martijn Faassen <faassen@startifact.com> wrote:
Also, is there any caching for already processed packages in the include finder code? If no, I'd probably like to contribute some, if I'll use z3c.autoinclude. :)
Ah, you're thinking in the same direction. I don't think there's any caching at all yet in autoinclude and that'd be the first thing I'd look at if I were to look into speeding it up. It's just I hadn't run into the pain point yet.
That's correct, there's no caching yet. I haven't done any profiling but I'm pretty sure that the pain-point algorithm is z3c.autoinclude.utils.dependencyForDottedName. Some sort of caching ought be a huge help there; the function currently iterates over all sys.path entries for each <autoincludeDependencies> directive found. Specifically my guess is that utils.py's L81-85 is what would benefit most from a cache, because that's where autoinclude actually peers into the sys.path entries to look for packages.
I'd really appreciate it if you could work with Dan to get performance data on autoinclude.
I'd be happy to! :) Dan, what do you think? The big reason I haven't tried any optimizations yet is that I haven't found a situation with noticeable slowdown that would let me measure any potential optimizations meaningfully, but it sounds like you've already "solved" that "problem." :) So I think the next step would be to reproduce that situation and measure the execution time before and after we add that caching strategy. Or we could just profile the code rather than relying on my guess. Regards, Ethan
Hi
exclude directiveinto zope.configuration.
Hey,
Dan Korostelev wrote:
The "exclude" directive provided by zc.configuration package is easy to use and straightforward. I think it's used almost in every zope-based application setup.
I highly doubt so; I don't find myself using it a lot myself, for instance. :)
Its implementation is very small and fits great in zope.configuration's standard directives. So I'd like to propose to move it to zope.configuration and make it always available for ZMCL files, just like the "include" directive, so people would'nt need to install extra package and include an extra "meta.zcml" file before being able to use it.
+1
+1
(note though that including an extra "meta.zcml" can be avoided if you make use of the z3c.autoinclude library)
Oh, cool. Now we only need to find out how to write an z3c.autoexlude and a z3c.autooverride library ;-) Regards Roger Ineichen
Regards,
Martijn
2009/2/26 Roger Ineichen <dev@projekt01.ch>:
(note though that including an extra "meta.zcml" can be avoided if you make use of the z3c.autoinclude library)
Oh, cool.
Now we only need to find out how to write an z3c.autoexlude and a z3c.autooverride library ;-)
:-)) +1 -- WBR, Dan Korostelev
Hi there, On Thu, Feb 26, 2009 at 12:43 PM, Roger Ineichen <dev@projekt01.ch> wrote:
Now we only need to find out how to write an z3c.autoexlude and a z3c.autooverride library ;-)
:) Just so that this doesn't get misinterpreted, autoinclude includes packages just like manual include statements do underneath, so there should be no incompatibility with the exclude directive at all. You can override autoincluded packages just like you'd override any included package. In addition there's an includeDependenciesOverrides directive which can include overrides automatically if you wish to do that. If a package uses z3c.autoinclude to include its dependencies there'll be no difference in what you can do concerning overrides or exclusion. Regards, Martijn
Hi Martijn
Betreff: Re: [Zope-dev] Proposal: merge zc.configuration's exclude directiveinto zope.configuration.
Hi there,
On Thu, Feb 26, 2009 at 12:43 PM, Roger Ineichen <dev@projekt01.ch> wrote:
Now we only need to find out how to write an z3c.autoexlude and a z3c.autooverride library ;-)
:)
Just so that this doesn't get misinterpreted, autoinclude includes packages just like manual include statements do underneath, so there should be no incompatibility with the exclude directive at all.
You can override autoincluded packages just like you'd override any included package. In addition there's an includeDependenciesOverrides directive which can include overrides automatically if you wish to do that.
If a package uses z3c.autoinclude to include its dependencies there'll be no difference in what you can do concerning overrides or exclusion.
I was just kiding as you saw. But this sounds very good to me since it has no other side effects! Thanks Roger Ineichen
Regards,
Martijn
On Feb 26, 2009, at 5:26 AM, Dan Korostelev wrote:
Hi there.
The "exclude" directive provided by zc.configuration package is easy to use and straightforward. I think it's used almost in every zope-based application setup. Its implementation is very small and fits great in zope.configuration's standard directives. So I'd like to propose to move it to zope.configuration and make it always available for ZMCL files, just like the "include" directive, so people would'nt need to install extra package and include an extra "meta.zcml" file before being able to use it.
If noone objects, I'd like to do that
That would be great. Thanks.
and make the "exclude" directive from zc.configuration point to the zope.configuration's implementation making the original place deprecated (however I guess the whole zc.configuration package should't be deprecated as it's intended to be a common place for configuration extensions, even if it has only one directive now).
Jim, if you're fine with that, can you please give me rights for zc.configuration on PYPI, my user name is "nadako".
Done, although, if you were me, you'd just leave it. :) Jim -- Jim Fulton Zope Corporation
2009/2/26 Jim Fulton <jim@zope.com>:
and make the "exclude" directive from zc.configuration point to the zope.configuration's implementation making the original place deprecated (however I guess the whole zc.configuration package should't be deprecated as it's intended to be a common place for configuration extensions, even if it has only one directive now).
Jim, if you're fine with that, can you please give me rights for zc.configuration on PYPI, my user name is "nadako".
Done, although, if you were me, you'd just leave it. :)
Thanks. I think we need to make people know that they don't need this package anymore if they are looking for a way to exclude configuration. -- WBR, Dan Korostelev
participants (5)
-
Dan Korostelev -
Ethan Jucovy -
Jim Fulton -
Martijn Faassen -
Roger Ineichen