Thanks, It was never my intention to imply that throwing $$$dollars at DC was required to improve scalability. What I said was based on perception of such requirement. I'll admit "currently" Zope does not scale as well or as elegantly as I would like. A statement I was making was that "if" I (or anyone) "believed" that scalability == $$$dollars, I would rather give to DC than Oracle. I believe Larry is doing all right on his own. :) I find Oracle pricing obscene. For the kind of $dollars it would take to license Oracle I would rather see what DC could do with the additional employees on beefing up Zope/ZODBs concurrency/scalabilities. There is a thread on the WebDB forum about a group of people talking about putting together a co-op to buy an Oracle license based on Oracles "unit" pricing scheme for a moderately powered single processor Intel machine it was going to run about $20,000. You would have to renegotiate if your site got busy and you added a processor. Yuck! I believe Zope's scalability issues will be solved over time regardless of external $dollars. I believe more $dollars could speed up the process, however it will still take some time. Currently ZEO is an option but I don't know how well understood it is by the community. Scalability Now can still be a sticky wicket. Future things on Zope's side: 1. Moore's law. 2. Advances in Python and it's performance. a. Python becomes fully multi-threaded, multi-processorable. b. Stackless Python gets adopted and performance improves. c. Viper 3. Advances in Zope, it's architecture and performance. a. Sam Rushing works miracles on Medusa/Zserver using Stackless Python and Continuations. b. Zope does process pooling with each process on a different processor. ZEO? c. ZODB does multiple data.fs's based on user design based on url path directories allowing different Zope processes on different processors to handle their respective dir. It would be nice if some C/Python programming guru/genius would solve Python's global threadlock issues where Python would scale on multiprocessor machines. This would potentially render scalability a moot point. The geniuses at AOLserver rewrote the threading of the built-in Tcl interpreter so that it would be multi-threaded and work properly with their nice multi-threaded webserver. Tcl's standard threading I don't think (I could be wrong) is as nice as Python's. If I had Oracle kind of money to throw around I would like to see Python's thread issue resolved where Python becomes fully multi-processorable. That would transparently scale Zope and benefit other Python user's as well. It would enable Python/Zope to embed well and play nice in multi-threaded webservers like AOLserver and others. IIRC Apache 2.0 will be multi-threaded. If Python scaled on multi-processors even a little guy could put together a dual-celery machine a serve a nice site. I agree wholeheartedly with you that there should be a concerted coordinated effort between DC and the community to improve scalability by allowing people to throw hardware at it. That was the primary issue causing me to wander the websphere looking at options. I've learned some things in the search, but I always looked back at Zope, even tho' my wife liked Enhydra's logo, she thinks the otter is cute. Zope's current logo is much nicer than the Purple and Gold. :) If ZEO currently scales well, then DC needs some testimonials on their website. There definitely needs to be some documentation best ways to... As you said, there is a consistent recurring thread of is ZServer alone or ZServer behind Apache better. Should I use a RDBMS or ZODB. ... Some of us have no legacy issues. From scratch what's the best way to design... I agree the solving of Performance/Scalability and Documentation issues would definitely help convert lookers into believers. I am currently working my way through the docs and plan on helping there. While I was wandering, in my heart Zope was the project I wanted to see succeed. I decided to put my mind where my heart was. Possibly someday some of my money too. :) Some of your other concerns I can't address. However, one of the nice things about Zope is that it is written in Python. Python imminently supports playing and algorithm improvement. :) Jimmie Houchin Patrick Phalen wrote:
[Jimmie Houchin, on Sat, 04 Mar 2000]
:: As a prodigal son here, I embraced Bobo/Zope early, became concerned :: about scalability, wandered the websphere, and have come back home. I :: have looked at Java/Servlets/Enhydra, AOLserver/ACS, etc. I decided to :: follow my heart. I like Python and Zope. I like the Python and Zope :: communities. This is where I want to make a difference and contribute. <snip> :: If I believed throwing $$$dollars to solve scalability was required. I :: would much rather throw $$$dollars at Digital Creations. Instead of a :: $20,000 to $500,000 on up license of Oracle which is per machine, per :: website, and potentially expires. Instead of requiring buying big, big :: hardware for $$$dollars. Throw the money towards Digital Creations or :: other Zope or Python developers to solve the issues which hold :: Zope/Python back. In the end you will have helped build a bigger, better :: open source software solution.
Welcome back, Jimmie. I enjoyed reading your analysis and agree with the thrust of it.
One thing you express, and which seems to have ossified into the current received wisdom, is that scalability will be improved when some mythical deep pockets consulting client pays to get it improved ; then everyone else will benefit. The problem is with the leap of faith it requires from the client.
I currently have several clients who are very interested in Zope. Their web sites already receive megahits. They are willing to pay money for consulting, but certainly won't bank their businesses on the chance that a magic key to scalability is found. As a consequence, the greatest commitment they will make is to install Zope on a Linux server on a VPN and "see how it goes" in a toy setting.
Therefore, I can testify that right now, today, Zope is being hurt by the lack of a story on scalability.
If economics or policy dictates that Zope requires an "Angel" to bankroll scalability research, we all have to be prepared for a scenario in which that Angel never appears. Personally, I don't like taking such a passive approach.
Two clear impediments to Zope's wide acceptance have been frequently discussed on this list -- documentation and performance.
The documentation "story" has travelled on an arc which might be expressed like this:
* Zope is Open Source. The community should document. * The community can't take on the deeper Zen issues * Volunteers from the community are time constrained * DC hires a technical writer * The ZDP is formed * ZDP suffers from the lack of time available to its volunteers * DC commits Amos to attack the general documentation problem and to evangelize documentation
Documentation currently seems headed on a good track. What have we learned from this trajectory? We've all learned that the Zope experiment is different from other Open Source projects. It *needs* the cooperation of *both* the Zope community and Zope's authors "shooting at the same basket" to solve this.
I believe scalability requires a similar coordinated attack and that it can't wait to be bankrolled.
ZEO, although a bit of a black box to us at the moment, sounds, on paper, like a logical and credible attack on one aspect of scalability -- the "throw hardware at it" approach.
No doubt there are other hidden opportunities. What kinds of hidden opportunities? Who knows; they're hidden. ;>) But can we bring them out of hiding?
Some of them are simple applications of available compute science, yet I don't see these codified or set down anywhere. Perhaps we should have a single HOW-TO repository for techniques and metrics people have discovered, or just happen to know, in the performance arena. This would be a FAQ along the lines of "I want to build my Zope site for best performance. What things should I do? What things should I avoid?" God knows this question comes up often enough.
Then there's the design and makeup of the Zope software itself. Its original design requirement probably didn't put scalability at the top of the stack. Here, there are probably other areas where the user community can't know enough about inner workings and overall design to contribute effectively -- but which require attention and a commitment from the folks at DC. I'll mention a couple of things that pop into my head. These are simply conjectural -- dialog starters. I'll be happy to be shown that I'm dead wrong about any of these conjectures.
Zope began as a brilliant notion, of Jim Fulton's devising. It began as a fews lines of code and has evolved to include better than two megabytes of code. DC has evolved from a company with one employee to a company with maybe a dozen engineers contributing to the code base and is looking to hire twenty more engineers.
Among this team of top notch talent, who takes ultimate responsibility for evangelizing and improving performance? Who leads the effort at code review, with an eye to marshalling refactoring efforts?
What would be an example? OK, Devil's Advocate: Here's one area of concern I have, perhaps unfounded. Digital Creations utilizes Use Cases as a tool for design, in the Ivor Jacobson tradition. Use Cases have many proponents and some detractors. One detractor is Bertrand Meyer, author of the classic _Object-Oriented Software Construction_. His beef with the use of Use Cases is that it poses a fundamental risk: it encourages a functional approach, based on processes (actions). He sees this as the reverse of O-O decomposition, which focuses on data abstraction. He believes it carries a risk of reverting, under the heading of object-oriented development, to the traditional forms of functional design, by considering what the system does, rather than what it does it to. He believes that, due to this risk, Use Cases should be confined to use as a validation tool, not a design tool, to avoid closing interfaces prematurely.
So let's say that Use Cases is a useful tool, but one that has a subtle built-in gotcha. What are the dangers? Naturally, one is negative impact on code maintainability/reusability.
Another may arise due to the need to balance two competing technical concerns: the desire to encapsulate abstractions, and the need to make certain abstractions visible to other modules. With regard to the dynamics of subprogram calls, the placement of declarations within modules can greatly affect the locality of reference and thus the paging behavior of a virtual memory system. Poor locality happens when subprogram calls occur across segments and lead to cache misses and page thrashing, which can ultimately slow down the whole system.
Is Zope prey to this? I don't have a clue. Does anyone? Like I say, I'd be happy to be shown that this concern is unfounded. But this is the sort of creeping problem which can enter into a growing code base like Zope's, as it is worked on by numerous people, each devoted on one module or interface. Combatting such problems probably requires having someone (someone blessed with both Jim Fulton's dedication to O-O principles and an unbending zealotry for absolute performance) committed to periodically flying above the code at a sufficient distance to see such patterns and making bombing runs against the target problems. As busy as everyone at DC is, I'm not certain that anyone is doing that at the moment.
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )