[Zope] Performance and Concurrency (Repost)

[Michel Pelletier]

Cary O'Brien wrote:
> 
> [I'm posting this again because, suprisingly, there was no response]
> 
> I was reading the thread about Zope vs AOLServer and performance
> and threading and all that.
> 
> I've done some work with other interpreters embedded in the apache
> child processes (tcl, mod-perl) so I started wondering if this would
> work for zope.  Somewhere I think there is a Python interpreter
> embedded a'la mod-perl. 

There is.

> If not, I'm pretty sure it can be done
> without too much grief, using mod-perl and mod-dtcl as starting
> points or examples.
> 
> It seems to me this would allow utilization of multiple processors,
> esp for apps with a database backend, since the work would be split up
> over n frontend apache server processes and n backend database
> processes (at least for oracle an PostgreSQL)[1].  All without
> worrying about threading...
> 
> But...
> 
> What happens when multiple processes attempt to access the zame ZODB
> Data.fs file? 

One succeeds, the rest fail.

> Is it multi-process safe? 

(I presume by 'it' you mean Zope)

Zope is, it's default storage, FileStorage, is not.

> Thread safe? 

Yes.

> How big are
> the sections protected by the mutexes? (or more importantly how long
> do they take to execute)?

Well, that question can't really be answered.  First off, there is a
global python interpreter lock.  I don't really want to get into the
details here, but basicly only one thread of execution can be in the
interpreter at once.  This is a pretty major limitation in Python's most
popular implementation, CPython (which is what Zope is based on).

The interpreter will 'give up the lock' every N bytecodes or so, there
is some mechanism for you to control this.  Check the python site for
more details.

Your idea is not bad, as long as you use a Storage that is multi-process
safe like ZEO.  I'm not sure if having as many Zope instances as Apache
instances is necessarily a good idea, since Zope instances can become
huge, many times bigger than their Apache counterparts.  Given Apache's
nature to do lots of forking and pre-forking (yes, this can be
controlled) this might lead to immense memory requirements.

-Michel