[Zope-dev] Proposal: merge zc.configuration's exclude directive into zope.configuration.

[Ethan Jucovy]

On Mon, Mar 2, 2009 at 9:59 AM, Martijn Faassen <faassen at startifact.com> wrote:
>> Also, is there any caching for already processed packages in the
>> include finder code? If no, I'd probably like to contribute some, if
>> I'll use z3c.autoinclude. :)
>
> Ah, you're thinking in the same direction. I don't think there's any
> caching at all yet in autoinclude and that'd be the first
> thing I'd look at if I were to look into speeding it up. It's just I
> hadn't run into the pain point yet.

That's correct, there's no caching yet.  I haven't done any profiling
but I'm pretty sure that the pain-point algorithm is
z3c.autoinclude.utils.dependencyForDottedName.  Some sort of caching
ought be a huge help there; the function currently iterates over all
sys.path entries for each <autoincludeDependencies> directive found.
Specifically my guess is that utils.py's L81-85 is what would benefit
most from a cache, because that's where autoinclude actually peers
into the sys.path entries to look for packages.

> I'd really appreciate it if you could work with Dan to get performance
> data on autoinclude.

I'd be happy to! :)

Dan, what do you think?  The big reason I haven't tried any
optimizations yet is that I haven't found a situation with noticeable
slowdown that would let me measure any potential optimizations
meaningfully, but it sounds like you've already "solved" that
"problem." :)  So I think the next step would be to reproduce that
situation and measure the execution time before and after we add that
caching strategy.  Or we could just profile the code rather than
relying on my guess.

Regards,
Ethan