[Zope] Scalability bombing missions (Was: Re: [Zope] ArsDigita .. request for comments)

[Jimmie Houchin]

Thanks,

It was never my intention to imply that throwing $$$dollars at DC was
required to improve scalability. What I said was based on perception of
such requirement. I'll admit "currently" Zope does not scale as well or
as elegantly as I would like. A statement I was making was that "if" I
(or anyone) "believed" that scalability == $$$dollars, I would rather
give to DC than Oracle. I believe Larry is doing all right on his own.
:)

I find Oracle pricing obscene. For the kind of $dollars it would take to
license Oracle I would rather see what DC could do with the additional
employees on beefing up Zope/ZODBs concurrency/scalabilities. There is a
thread on the WebDB forum about a group of people talking about putting
together a co-op to buy an Oracle license based on Oracles "unit"
pricing scheme for a moderately powered single processor Intel machine
it was going to run about $20,000. You would have to renegotiate if your
site got busy and you added a processor. Yuck!

I believe Zope's scalability issues will be solved over time regardless
of external $dollars. I believe more $dollars could speed up the
process, however it will still take some time. Currently ZEO is an
option but I don't know how well understood it is by the community.

Scalability Now can still be a sticky wicket.

Future things on Zope's side:
1. Moore's law.
2. Advances in Python and it's performance.
   a. Python becomes fully multi-threaded, multi-processorable.
   b. Stackless Python gets adopted and performance improves.
   c. Viper 
3. Advances in Zope, it's architecture and performance.
   a. Sam Rushing works miracles on Medusa/Zserver using Stackless
Python and Continuations.
   b. Zope does process pooling with each process on a different
processor. ZEO?
   c. ZODB does multiple data.fs's based on user design based on url
path directories
        allowing different Zope processes on different processors to
handle their respective dir.

It would be nice if some C/Python programming guru/genius would solve
Python's global threadlock issues where Python would scale on
multiprocessor machines. This would potentially render scalability a
moot point. The geniuses at AOLserver rewrote the threading of the
built-in Tcl interpreter so that it would be multi-threaded and work
properly with their nice multi-threaded webserver. Tcl's standard
threading I don't think (I could be wrong) is as nice as Python's.

If I had Oracle kind of money to throw around I would like to see
Python's thread issue resolved where Python becomes fully
multi-processorable. That would transparently scale Zope and benefit
other Python user's as well. It would enable Python/Zope to embed well
and play nice in multi-threaded webservers like AOLserver and others.
IIRC Apache 2.0 will be multi-threaded. If Python scaled on
multi-processors even a little guy could put together a dual-celery
machine a serve a nice site.

I agree wholeheartedly with you that there should be a concerted
coordinated effort between DC and the community to improve scalability
by allowing people to throw hardware at it. That was the primary issue
causing me to wander the websphere looking at options. I've learned some
things in the search, but I always looked back at Zope, even tho' my
wife liked Enhydra's logo, she thinks the otter is cute. Zope's current
logo is much nicer than the Purple and Gold. :)

If ZEO currently scales well, then DC needs some testimonials on their
website.

There definitely needs to be some documentation best ways to...
As you said, there is a consistent recurring thread of is ZServer alone
or ZServer behind Apache better.
Should I use a RDBMS or ZODB.
...

Some of us have no legacy issues. From scratch what's the best way to
design...

I agree the solving of Performance/Scalability and Documentation issues
would definitely help convert lookers into believers.

I am currently working my way through the docs and plan on helping
there.

While I was wandering, in my heart Zope was the project I wanted to see
succeed. I decided to put my mind where my heart was.  Possibly someday
some of my money too. :)

Some of your other concerns I can't address. However, one of the nice
things about Zope is that it is written in Python. Python imminently
supports playing and algorithm improvement. :)

Jimmie Houchin



Patrick Phalen wrote:
> 
> [Jimmie Houchin, on Sat, 04 Mar 2000]
> 
> :: As a prodigal son here, I embraced Bobo/Zope early, became concerned
> :: about scalability, wandered the websphere, and have come back home. I
> :: have looked at Java/Servlets/Enhydra, AOLserver/ACS, etc. I decided to
> :: follow my heart. I like Python and Zope. I like the Python and Zope
> :: communities. This is where I want to make a difference and contribute.
> <snip>
> :: If I believed throwing $$$dollars to solve scalability was required. I
> :: would much rather throw $$$dollars at Digital Creations. Instead of a
> :: $20,000 to $500,000 on up license of Oracle which is per machine, per
> :: website, and potentially expires. Instead of requiring buying big, big
> :: hardware for $$$dollars. Throw the money towards Digital Creations or
> :: other Zope or Python developers to solve the issues which hold
> :: Zope/Python back. In the end you will have helped build a bigger, better
> :: open source software solution.
> 
> Welcome back, Jimmie. I enjoyed reading your analysis and agree with
> the thrust of it.
> 
> One thing you express, and which seems to have ossified into the current
> received wisdom, is that scalability will be improved  when some
> mythical deep pockets consulting client pays to get it improved ; then
> everyone else will benefit. The problem is with the leap of faith it
> requires from the client.
> 
> I currently have several clients who are very interested in Zope.
> Their web sites already receive megahits. They are willing to pay money
> for consulting, but certainly won't bank their businesses on the chance
> that a magic key to scalability is found. As a consequence, the
> greatest commitment they will make is to install Zope on a Linux server
> on a VPN and "see how it goes" in a toy setting.
> 
> Therefore, I can testify that right now, today, Zope is being hurt by
> the lack of a story on scalability.
> 
> If economics or policy dictates that Zope requires an "Angel" to
> bankroll scalability research, we all have to be prepared for a
> scenario in which that Angel never appears. Personally, I don't like
> taking such a passive approach.
> 
> Two clear impediments to Zope's wide acceptance have been frequently
> discussed on this list -- documentation and performance.
> 
> The documentation "story" has travelled on an arc which might be
> expressed like this:
> 
> * Zope is Open Source. The community should document.
> * The community can't take on the deeper Zen issues
> * Volunteers from the community are time constrained
> * DC hires a technical writer
> * The ZDP is formed
> * ZDP suffers from the lack of time available to its volunteers
> * DC commits Amos to attack the general documentation problem and to
> evangelize documentation
> 
> Documentation currently seems headed on a good track. What have we
> learned from this trajectory? We've all learned that the Zope experiment
> is different from other Open Source projects. It *needs* the cooperation
> of *both* the Zope community and Zope's authors "shooting at the same
> basket" to solve this.
> 
> I believe scalability requires a similar coordinated attack and that it
> can't wait to be bankrolled.
> 
> ZEO, although a bit of a black box to us at the moment, sounds, on
> paper, like a logical and credible attack on one aspect of scalability
> -- the "throw hardware at it" approach.
> 
> No doubt there are other hidden opportunities. What kinds of hidden
> opportunities? Who knows; they're hidden. ;>) But can we bring them out
> of hiding?
> 
> Some of them are simple applications of available compute science, yet
> I don't see these codified or set down anywhere. Perhaps we should have
> a single HOW-TO repository for techniques and metrics people have
> discovered, or just happen to know, in the performance arena. This
> would be a FAQ along the lines of "I want to build my Zope site for
> best performance. What things should I do? What things should I avoid?"
> God knows this question comes up often enough.
> 
> Then there's the design and makeup of the Zope software itself. Its
> original design requirement probably didn't put scalability at the top
> of the stack. Here, there are probably other areas where the user
> community can't know enough about inner workings and overall design to
> contribute effectively -- but which require attention and a commitment
> from the folks at DC. I'll mention a couple of things that pop into my
> head. These are simply conjectural -- dialog starters. I'll be happy to
> be shown that I'm dead wrong about any of these conjectures.
> 
> Zope began as a brilliant notion, of Jim Fulton's devising. It began as
> a fews lines of code and has evolved to include better than two
> megabytes of code. DC has evolved from a company with one employee to a
> company with maybe a dozen engineers contributing to the code
> base and is looking to hire twenty more engineers.
> 
> Among this team of top notch talent, who takes ultimate responsibility
> for evangelizing and improving performance? Who leads the effort at
> code review, with an eye to marshalling refactoring efforts?
> 
> What would be an example? OK, Devil's Advocate: Here's one area of
> concern I have, perhaps unfounded. Digital Creations utilizes Use Cases
> as a tool for design, in the Ivor Jacobson tradition. Use Cases have
> many proponents and some detractors. One detractor is Bertrand Meyer,
> author of the classic _Object-Oriented Software Construction_. His beef
> with the use of Use Cases is that it poses a fundamental risk: it
> encourages a functional approach, based on processes (actions). He sees
> this as the reverse of O-O decomposition, which focuses on data
> abstraction. He believes it carries a risk of reverting, under the
> heading of object-oriented development, to the traditional forms of
> functional design, by considering what the system does, rather than
> what it does it to. He believes that, due to this risk, Use Cases
> should be confined to use as a validation tool, not a design tool, to
> avoid closing interfaces prematurely.
> 
> So let's say that Use Cases is a useful tool, but one that has a subtle
> built-in gotcha. What are the dangers? Naturally, one is negative
> impact on code maintainability/reusability.
> 
> Another may arise due to the need to balance two competing technical
> concerns: the desire to encapsulate abstractions, and the need to make
> certain abstractions visible to other modules. With regard to the
> dynamics of subprogram calls, the placement of declarations within
> modules can greatly affect the locality of reference and thus the
> paging behavior of a virtual memory system. Poor locality happens when
> subprogram calls occur across segments and lead to cache misses and
> page thrashing, which can ultimately slow down the whole system.
> 
> Is Zope prey to this? I don't have a clue. Does anyone? Like I say, I'd
> be happy to be shown that this concern is unfounded. But this is the
> sort of creeping problem which can enter into a growing code base like
> Zope's, as it is worked on by numerous people, each devoted on one
> module or interface. Combatting such problems probably requires having
> someone (someone blessed with both Jim Fulton's dedication to O-O
> principles and an unbending zealotry for absolute performance)
> committed to periodically flying above the code at a sufficient
> distance to see such patterns and making bombing runs against the
> target problems. As busy as everyone at DC is, I'm not certain that
> anyone is doing that at the moment.
> 
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )