[ZODB-Dev] RE: [Zope] Re: Squid headaches

[Barry Warsaw]

On Sat, 2003-05-31 at 13:42, Dieter Maurer wrote:

> I do not yet fully understand why, but the complete host seems to
> be affected by the high disc bandwidth required during packing.

Note that with Berkeley storages, one of the biggest boosts to
performance you can get is by separating the BerkeleyDB database and log
files to different disks.  This is a low-level BerkeleyDB configuration
option that's exposed in the storage configuration parameters. 
FileStorage doesn't have this "luxury".

> In the second packing phase (the so called copying phase),
> the packing thread helds the storage lock most of the time.
> This can make accessing the ZODB impossible by normal operations,
> especially because the "lock_release", "lock_acquire" probably
> does not succeed in giving the lock to a waiting thread.
> I have a patch that restricts this locking to only a minor
> fraction of time.

Berkeley storages release the storage lock during the various phases of
packing.  It's not terribly granular, but it would give opportunities
for other work to happen during packing.  That, and regular autopacking
might make things acceptably responsive.

-Barry