On Mon, Mar 2, 2009 at 9:59 AM, Martijn Faassen <faassen@startifact.com> wrote:
Also, is there any caching for already processed packages in the include finder code? If no, I'd probably like to contribute some, if I'll use z3c.autoinclude. :)
Ah, you're thinking in the same direction. I don't think there's any caching at all yet in autoinclude and that'd be the first thing I'd look at if I were to look into speeding it up. It's just I hadn't run into the pain point yet.
That's correct, there's no caching yet. I haven't done any profiling but I'm pretty sure that the pain-point algorithm is z3c.autoinclude.utils.dependencyForDottedName. Some sort of caching ought be a huge help there; the function currently iterates over all sys.path entries for each <autoincludeDependencies> directive found. Specifically my guess is that utils.py's L81-85 is what would benefit most from a cache, because that's where autoinclude actually peers into the sys.path entries to look for packages.
I'd really appreciate it if you could work with Dan to get performance data on autoinclude.
I'd be happy to! :) Dan, what do you think? The big reason I haven't tried any optimizations yet is that I haven't found a situation with noticeable slowdown that would let me measure any potential optimizations meaningfully, but it sounds like you've already "solved" that "problem." :) So I think the next step would be to reproduce that situation and measure the execution time before and after we add that caching strategy. Or we could just profile the code rather than relying on my guess. Regards, Ethan