[Grok-dev] Neanderthal-sprint: Speeding up the Grok startup with zc.configuration

[Kevin Smith]

Hi Lennart,

My experience matches your assessment.

My startup went from 3-4 secs (reasonable) to 35 seconds (suicidal) as 
more and more content was put in the zodb and a large memory footprint.. 
I can't seem to find a good link, but from what I understand this is 
because the zodb re-creates an index upon startup. As a result I 
switched to PGStorage and startup has dropped back to a flat 4.5 seconds 
with 1.6gb of content with a low memory footprint. AFAIK 
DirectoryStorage offers the same advantage and low memory footprint. 
Persistent caches are also an option, but I've had problems with those 
getting out of sync in the past and with large memory footprint issue.

Kevin Smith

Lennart Regebro wrote:
> On 9/30/07, Martijn Faassen <faassen at startifact.com> wrote:
>   
>> Hey,
>>
>> I'm now in Germany in the hotel! I thought of another sprint topic: Grok
>> startup time.
>>
>> It'd be quite interesting to see what we can do to increase startup time
>> of Grok. Jim has made a package available called zc.configure that can
>> be used to exclude ZCML from being loaded. This can, according to Jim,
>> be used to cut startup time (and also test setup time) significantly.
>>
>> This is quite interesting - I'd really love it if Grok could start up a
>> lot faster than it does now. There's also likely a lot of ZCML we can
>> cut out (anything doing the ZMI for starters!).
>>     
>
> Today me, Joachim Schmitz and Ahmet Toker tried to speed up the Grok
> startup by excluding zcml that isn't used in Grok. Basically, we used
> Jim Fultons new zc.configuration module to exclude browser modules
> that are loaded in Zope3. It would obviously be better if the
> separation between the functionality and the web pages used by the ZMI
> was higher, so we didn't have to include things from the beginning,
> but that's much harder work.
>
> After a bit of work we had found around 20 zcml-files that could be
> excluded without breaking the tests. That decreased the startup time
> with about 10%. There are probably more modules to exclude, but it's
> going to take a while to go through all of zope3 that way, and we
> don't know how much faster it will be. There are no more than 100
> modules all in all, and not all of them have a browser.zcml or browser
> module. I doubt that we'll find more than 40 zcml all in all, which
> could cut startup time with maybe 20% all in all, whoch s not very
> impressive. When startup time already is in the single-seconds on a
> reasonably modern computer, cutting off a couple of tenths, or maybe a
> second, doesn't really seem worth the hassle and the risk of breaking
> stuff.
>
> And besides, Grok is currently just a couple of tenths slower than a
> completely empty Zope2 site, and we know that what takes long when
> starting Zope2 is not Zope2 itself, but the application. A typical
> Plone 2.5 site takes 3 times as long as an empty Zope 2.9 site,
> meaning that in real life with Grok even 30% speedup of Grok startup
> might only mean 10% if you use a large application like a CMS. Is it
> worth it?
>
> The work we did can be found here:
> http://svn.zope.org/grok/branches/neanderthal-startupspeed/
>
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/grok-dev/attachments/20071002/8ee8701b/attachment.htm