[Zope] Scalability bombing missions

[Jimmie Houchin]

Paul Everitt wrote:
> Hi Jimmie and Patrick.
> 
> Jimmie wrote:
[snip]
> > I find Oracle pricing obscene. For the kind of $dollars it
> 
> Architectures like Oracle with Vignette running atop it are very
> capital-intensive.  Something that startup dot coms don't have a lot of.
> When you've spent all your budget on licenses, and _then_ you start
> writing your application...

Yes. Not very enabling for the idea guy.

[snip]
> > I believe Zope's scalability issues will be solved over time
> > regardless
> > of external $dollars. I believe more $dollars could speed up the
> > process, however it will still take some time. Currently ZEO is an
> > option but I don't know how well understood it is by the community.
> 
> It will certainly take some time.  But obvious things can be solved as a
> higher priority.  Since scalability means different things to different
> people, though, there isn't One Answer.
> 
> > Scalability Now can still be a sticky wicket.
> >
> > Future things on Zope's side:
> > 1. Moore's law.
> > 2. Advances in Python and it's performance.
> >    a. Python becomes fully multi-threaded, multi-processorable.
> >    b. Stackless Python gets adopted and performance improves.
> >    c. Viper
> 
> I don't feel that advances in Python are absolutely required.  Apache
> doesn't have a threaded model (before v2), and if Zope could dish out
> requests like Apache 1.x, everybody would be awfully happy. :^)

Woohoo! Absolutely. :)
However, if a multi-threaded/multiprocessor capable Python were
available to embed into AOLserver then it seems to me that the improved
performance would be transparent.

> This begs the question: how fast is fast enough?  These stupid PCWeek
> tests, that give MTS a higher performance score than WebObjects because
> it claims to do 4,000 requests/sec vs. 2,000 requests/sec, are just
> ridiculous.  At 150 requests/sec, you're talking about the top one
> percent of websites on the Web.

This is true if the requests are evenly amortized throughout the 24 hour
period. Unfortunately they aren't. For me fast enough means that your
machine is not maxed out (or croaking) when handling peak loads. Your
visitors handle the speed/latency well. It is all very relative.

> The other worry is that these super-freaky Python advances will require
> cooperation from the app developers to keep the train on the tracks.

I agree. I was just hypothesizing. I would generally prefer to keep with
the advancements in core Python. Who knows what Py3k will bring. :)

> > 3. Advances in Zope, it's architecture and performance.
> >    a. Sam Rushing works miracles on Medusa/Zserver using Stackless
> > Python and Continuations.
> 
> Again, I'm not so sure this is the big answer.  I don't think there's a
> lot of evidence that Medusa/ZServer is the primary bottleneck.

True and maybe not. However as Zope pushes into heavier trafficked
environments who knows. But we'll take whatever speed gains we can. :)

> >    b. Zope does process pooling with each process on a different
> > processor. ZEO?
> 
> Yep, this is ultimately an attractive architecture.  Particularly when
> you pour all three tiers (presentation, logic, data) into it, you get a
> single facility for scaling and distributing all three.

What I like about this is the capability of distributing your work load
regardless of single machine or cluster.

> >    c. ZODB does multiple data.fs's based on user design based on url
> > path directories
> 
> Right, though I'd encourage you to instead speak in terms of the goal,
> rather than the solution.  Is the goal here to solve the 2Gb problem on
> Linux/x86, to "limit corruption", or is it to do resource accounting on
> a per user basis?  As Jim has pointed out, if your goal is the latter,
> you won't get there -- user data can still appear in global things like
> catalogs.

Actually not necessarily either. I don't want data corruption, so if
that's an issue it will need to be addressed. Say for example I have a
very large database. Like this:

ZAuto
ZAuto/parts/
ZAuto/manufacturers/
ZAuto/distributors/
ZAuto/distributors/...(orders, address, preferences, etc.)
ZAuto/resellers/...(orders, address, preferences, etc.)
...

Let's say that ZAuto/parts is a 5gb directory which is reasonably stable
updated monthly.
ZAuto/manufacturers isn't so large but is also stable with periodic
updates.
ZAuto/distributors and ZAuto/resellers are much more dynamic with
changes and additions.

ZAuto/parts could be optimized for search performance. It only needs to
be backed up periodically when bulk changes are made. It will relatively
almost never need packed. It could sit on it's own drive, own machine,
etc.

It would be nice to be able to partition the database into
stable/volatile files. Since the stable partition is also the largest,
it seems this would also help performance of the more dynamic section.

> Jim's counterproposal, which he claims is easier than doing accounting
> across separate Data.fs files, is to finish putting accounting hooks
> into the object database.  Whenever a user exceeded their capacity (no
> matter where the object), an exception would be raised.  This would
> probably require that people be able to do their own packing.
> 
> I gotta admit, I'm awfully intrigued by this.  It is the kind of thing
> that is _seriously_ compelling vs. other architectures, where you have
> to keep on-disk quotas for presentation and logic, and in-RDBMS quotas
> for data.

My example above doesn't fit into this situation, at least as far as I
can tell. There are no user quotas or explicit restrictions.

> Still, the point is, make sure you speak in terms of goals first.
> 
> >         allowing different Zope processes on different processors to
> > handle their respective dir.
> >
> > It would be nice if some C/Python programming guru/genius would solve
> > Python's global threadlock issues where Python would scale on
> > multiprocessor machines. This would potentially render scalability a
> 
> Doubt it.  There's still so much more, such as data scalability (vs. raw
> performance).  As for the latter, we at DC _still_ need to make raw
> performance go up.
> 
> Personally, I think a very smart caching system (where subtemplates are
> pre-rendered in ZServer and know when the objects they depend on have
> changed) is a huge win for a significant portion of high-load
> situations.

I have somewhat lumped raw performance and scalability together. In many
ways they are hard to separate. I agree a smart caching system would be
nice and a tremendous boon to performance. In Philip's writings, he
believes that performance is best when the entire db resides in memory.
Sometimes that is not possible. However, if you cache the most often
accessed parts you reap tremendously.

> > moot point. The geniuses at AOLserver rewrote the threading of the
> > built-in Tcl interpreter so that it would be multi-threaded and work
> > properly with their nice multi-threaded webserver. Tcl's standard
> > threading I don't think (I could be wrong) is as nice as Python's.
> 
> Right.  Does this, though, limit them on the platforms you run on
> because of assumptions in the threading model?

I don't know, I am not a threading expert. However, with AOLserver, I
believe that stayed with the POSIX model. So if Python was such it seems
it wouldn't be a problem in most cases.

> > If I had Oracle kind of money to throw around I would like to see
> > Python's thread issue resolved where Python becomes fully
> > multi-processorable. That would transparently scale Zope and benefit
> > other Python user's as well. It would enable Python/Zope to embed well
> > and play nice in multi-threaded webservers like AOLserver and others.
> > IIRC Apache 2.0 will be multi-threaded. If Python scaled on
> > multi-processors even a little guy could put together a dual-celery
> > machine a serve a nice site.
> 
> Right.  But again, this begs the question: what is a "nice site"?  What
> if the goal of a "nice site" could be met w/out the disruption of
> rearchitecting Python?
> 
> Is a "nice site" 100K hits/day?  A million hits/day?  Ten million
> hits/day?
> 
> This steps into the next part of the problem.  To date, few in the Zope
> community has been willing to publish any numbers for highly-traffic'd
> sites, despite pleas even from our board chairman.  Perhaps that means
> there aren't really any sites where Zope's limit has been reached?
> Probably not.  But why are people so reticent to go on the record with
> their hit logs?
> 
> But it _does_ mean that, until folks are willing to come forward and
> provide numbers and test conditions, that we'll be working without a
> map.

Yes, this is a very subjective area. My original problem with Zope's
scalability and what would scare me with a site is the inability to
throw hardware at it. ZEO didn't exist then. The question exists though
is if a Zope based website was getting 1 millions hits a day and gets
Slashdotted or some other such phenomena and in a few weeks or months
grows to 5 or 10 million, what are their options within Zope? Where will
the bottlenecks come from? Can they throw hardware at it? What if
tomorrow my "Slamazon" bookstore went online and was an instant success?

> > I agree wholeheartedly with you that there should be a concerted
> > coordinated effort between DC and the community to improve scalability
> > by allowing people to throw hardware at it. That was the primary issue
> > causing me to wander the websphere looking at options. I've
> > learned some
> > things in the search, but I always looked back at Zope, even tho' my
> > wife liked Enhydra's logo, she thinks the otter is cute.
> > Zope's current
> > logo is much nicer than the Purple and Gold. :)
> >
> > If ZEO currently scales well, then DC needs some testimonials on their
> > website.
> 
> Right.
> 
> > There definitely needs to be some documentation best ways to...
> > As you said, there is a consistent recurring thread of is
> > ZServer alone
> > or ZServer behind Apache better.
> > Should I use a RDBMS or ZODB.
> 
> Right.
> 
> > Some of us have no legacy issues. From scratch what's the best way to
> > design...
> 
> Ditto.
> 
> > I agree the solving of Performance/Scalability and
> > Documentation issues
> > would definitely help convert lookers into believers.
> 
> That's a good way to phrase it.

Amen!

> --Paul

Jimmie Houchin