[Zope] Re: Zope 2 and Zope 3

[Geoff Gerrietts]

On 2/14/07, Philipp von Weitershausen <philipp at weitershausen.de> wrote:
>
> There will definitely be a Zope 2.11 and at this point I see no reason
> why there shouldn't be a Zope 2.12. We (the Zope Community, not
> necessarily Zope Corporation) will maintain Zope 2 as long as it's
> necessary. "Maintaining" in this case also means integrating more Zope 3
> technologies with each release. Nowadays (Zope 2.10) you can already
> write applications, or at least base libraries, that work on both Zope 3
> and Zope 2.

This is very helpful for setting a planning horizon, thank you.

> I'm surprised you're experiencing poor scaling. I think Zope can scale
> pretty well, or at least it can be *made* to scale pretty well. It all
> depends on your setup and, of course, on your application.

I can't swear to the accuracy of what I'm about to say but it is an
operating hypothesis that describes accurately behavior that we are
seeing.

Our application is highly dynamic, by which I mean that satisfying any
given request may require a great deal of database activity. The
database in question is almost always Oracle; I'm not counting the
ZODB accesses, here.

Oracle will handle a great many concurrent requests fairly gracefully,
but its average transaction time is not always excellent, especially
for moderately complicated queries. While most queries are sub-second,
populating a page that requires multiple queries to fill in all its
data fields, or even populating a page with only one or two
complicated queries, can require greater than a single second of
database I/O. This cost is offset by the ability to have other threads
processing while your first thread waits for the database to come
back.

Zope likes to have 4 threads processing requests. This preference is
documented in several places, and experiments have historically borne
out the "4 thread hypothesis". We have been unable to realize
significant performance improvements by increasing the number of
available worker threads, but we are able to realize performance
improvements by pointing two Zope servers against the same database.

Zope isn't using much CPU, and it's not using much memory, but it
peaks out at 4 concurrent requests; everything else goes in a queue
behind them. If those 4 concurrent requests happen to take 4 or 5
seconds each, the queue builds deep behind them, possibly with very
simple one-offs.

Historically, we've addressed this by sticking another Zope out there.
We have plans to run multiple Zope instances on a single machine, like
they do (did?) at plone.org; this still requires configuration
management work and testing. But the feeling is that we have a lot
more hardware than we really should need, to handle the volumes we're
dealing with, and we have that hardware because we need more
"channels" to handle our concurrent traffic load.

> As for source control, I figure all of your code (DTML, yuck) is in the
> ZODB. This went out of fashion a long time ago, most serious development
> happens on the filesystem (in Python packages) which can obviously be
> source-controlled very well.

A good deal of the code is in the ZODB, because it's (yuck) DTML
documents, yes. We have another sizable chunk that lives on the
filesystem. The challenge in managing the build/deploy process has
been trying to find a way to keep those two conjoined. I suspect that
any dependency on the ZODB at all is likely to be considered an
impediment, and my quick eyeballing says it's not gone or even really
optional under Zope 3 (but I'm sure we could work something out.)


> Regarding "oh you'll hafta start over", it's pretty much true, if you
> want to switch to Zope 3. But nobody says you have to. You can do it
> incrementally by porting some of your app's functionality to Zope 3
> components step by step (as mentioned already, most work in Zope 2). Big
> projects like Plone aren't rewriting their whole codebase either, but
> new development is completely Zope3-based. Their target platform still
> is Zope 2, though.

"You'll hafta start over" is only ever true in degrees, though, and
I'm still trying to figure out what the degrees are. The business
logic will still be the same. Can we hack together a DTML processor
that allows us to export the DTML documents to the filesystem and
publish them from there? Maybe, I don't know. Did someone else already
do that? Don't know that either. How dramatically will our products
need to change? Probably 75% of our Python code is written in a
bastardized form of ExternalMethod; we might be able to leverage that
unfortunate architectural choice to significant advantage during our
porting phase. These are the kinds of questions I have, and I think
the answers probably aren't easy or someone would have offered them
up.

Someone's going to need to learn enough Zope 3 to answer the
questions. I'm not sure it will be me, but maybe. :)

Thanks for all the insight, I'm making plans and pleas and I have a
direction for my research now. :)

Thanks,
--G.