Paul Everitt wrote:
Hi Jimmie and Patrick.
Jimmie wrote: [snip]
I find Oracle pricing obscene. For the kind of $dollars it
Architectures like Oracle with Vignette running atop it are very capital-intensive. Something that startup dot coms don't have a lot of. When you've spent all your budget on licenses, and _then_ you start writing your application...
Yes. Not very enabling for the idea guy. [snip]
I believe Zope's scalability issues will be solved over time regardless of external $dollars. I believe more $dollars could speed up the process, however it will still take some time. Currently ZEO is an option but I don't know how well understood it is by the community.
It will certainly take some time. But obvious things can be solved as a higher priority. Since scalability means different things to different people, though, there isn't One Answer.
Scalability Now can still be a sticky wicket.
Future things on Zope's side: 1. Moore's law. 2. Advances in Python and it's performance. a. Python becomes fully multi-threaded, multi-processorable. b. Stackless Python gets adopted and performance improves. c. Viper
I don't feel that advances in Python are absolutely required. Apache doesn't have a threaded model (before v2), and if Zope could dish out requests like Apache 1.x, everybody would be awfully happy. :^)
Woohoo! Absolutely. :) However, if a multi-threaded/multiprocessor capable Python were available to embed into AOLserver then it seems to me that the improved performance would be transparent.
This begs the question: how fast is fast enough? These stupid PCWeek tests, that give MTS a higher performance score than WebObjects because it claims to do 4,000 requests/sec vs. 2,000 requests/sec, are just ridiculous. At 150 requests/sec, you're talking about the top one percent of websites on the Web.
This is true if the requests are evenly amortized throughout the 24 hour period. Unfortunately they aren't. For me fast enough means that your machine is not maxed out (or croaking) when handling peak loads. Your visitors handle the speed/latency well. It is all very relative.
The other worry is that these super-freaky Python advances will require cooperation from the app developers to keep the train on the tracks.
I agree. I was just hypothesizing. I would generally prefer to keep with the advancements in core Python. Who knows what Py3k will bring. :)
3. Advances in Zope, it's architecture and performance. a. Sam Rushing works miracles on Medusa/Zserver using Stackless Python and Continuations.
Again, I'm not so sure this is the big answer. I don't think there's a lot of evidence that Medusa/ZServer is the primary bottleneck.
True and maybe not. However as Zope pushes into heavier trafficked environments who knows. But we'll take whatever speed gains we can. :)
b. Zope does process pooling with each process on a different processor. ZEO?
Yep, this is ultimately an attractive architecture. Particularly when you pour all three tiers (presentation, logic, data) into it, you get a single facility for scaling and distributing all three.
What I like about this is the capability of distributing your work load regardless of single machine or cluster.
c. ZODB does multiple data.fs's based on user design based on url path directories
Right, though I'd encourage you to instead speak in terms of the goal, rather than the solution. Is the goal here to solve the 2Gb problem on Linux/x86, to "limit corruption", or is it to do resource accounting on a per user basis? As Jim has pointed out, if your goal is the latter, you won't get there -- user data can still appear in global things like catalogs.
Actually not necessarily either. I don't want data corruption, so if that's an issue it will need to be addressed. Say for example I have a very large database. Like this: ZAuto ZAuto/parts/ ZAuto/manufacturers/ ZAuto/distributors/ ZAuto/distributors/...(orders, address, preferences, etc.) ZAuto/resellers/...(orders, address, preferences, etc.) ... Let's say that ZAuto/parts is a 5gb directory which is reasonably stable updated monthly. ZAuto/manufacturers isn't so large but is also stable with periodic updates. ZAuto/distributors and ZAuto/resellers are much more dynamic with changes and additions. ZAuto/parts could be optimized for search performance. It only needs to be backed up periodically when bulk changes are made. It will relatively almost never need packed. It could sit on it's own drive, own machine, etc. It would be nice to be able to partition the database into stable/volatile files. Since the stable partition is also the largest, it seems this would also help performance of the more dynamic section.
Jim's counterproposal, which he claims is easier than doing accounting across separate Data.fs files, is to finish putting accounting hooks into the object database. Whenever a user exceeded their capacity (no matter where the object), an exception would be raised. This would probably require that people be able to do their own packing.
I gotta admit, I'm awfully intrigued by this. It is the kind of thing that is _seriously_ compelling vs. other architectures, where you have to keep on-disk quotas for presentation and logic, and in-RDBMS quotas for data.
My example above doesn't fit into this situation, at least as far as I can tell. There are no user quotas or explicit restrictions.
Still, the point is, make sure you speak in terms of goals first.
allowing different Zope processes on different processors to handle their respective dir.
It would be nice if some C/Python programming guru/genius would solve Python's global threadlock issues where Python would scale on multiprocessor machines. This would potentially render scalability a
Doubt it. There's still so much more, such as data scalability (vs. raw performance). As for the latter, we at DC _still_ need to make raw performance go up.
Personally, I think a very smart caching system (where subtemplates are pre-rendered in ZServer and know when the objects they depend on have changed) is a huge win for a significant portion of high-load situations.
I have somewhat lumped raw performance and scalability together. In many ways they are hard to separate. I agree a smart caching system would be nice and a tremendous boon to performance. In Philip's writings, he believes that performance is best when the entire db resides in memory. Sometimes that is not possible. However, if you cache the most often accessed parts you reap tremendously.
moot point. The geniuses at AOLserver rewrote the threading of the built-in Tcl interpreter so that it would be multi-threaded and work properly with their nice multi-threaded webserver. Tcl's standard threading I don't think (I could be wrong) is as nice as Python's.
Right. Does this, though, limit them on the platforms you run on because of assumptions in the threading model?
I don't know, I am not a threading expert. However, with AOLserver, I believe that stayed with the POSIX model. So if Python was such it seems it wouldn't be a problem in most cases.
If I had Oracle kind of money to throw around I would like to see Python's thread issue resolved where Python becomes fully multi-processorable. That would transparently scale Zope and benefit other Python user's as well. It would enable Python/Zope to embed well and play nice in multi-threaded webservers like AOLserver and others. IIRC Apache 2.0 will be multi-threaded. If Python scaled on multi-processors even a little guy could put together a dual-celery machine a serve a nice site.
Right. But again, this begs the question: what is a "nice site"? What if the goal of a "nice site" could be met w/out the disruption of rearchitecting Python?
Is a "nice site" 100K hits/day? A million hits/day? Ten million hits/day?
This steps into the next part of the problem. To date, few in the Zope community has been willing to publish any numbers for highly-traffic'd sites, despite pleas even from our board chairman. Perhaps that means there aren't really any sites where Zope's limit has been reached? Probably not. But why are people so reticent to go on the record with their hit logs?
But it _does_ mean that, until folks are willing to come forward and provide numbers and test conditions, that we'll be working without a map.
Yes, this is a very subjective area. My original problem with Zope's scalability and what would scare me with a site is the inability to throw hardware at it. ZEO didn't exist then. The question exists though is if a Zope based website was getting 1 millions hits a day and gets Slashdotted or some other such phenomena and in a few weeks or months grows to 5 or 10 million, what are their options within Zope? Where will the bottlenecks come from? Can they throw hardware at it? What if tomorrow my "Slamazon" bookstore went online and was an instant success?
I agree wholeheartedly with you that there should be a concerted coordinated effort between DC and the community to improve scalability by allowing people to throw hardware at it. That was the primary issue causing me to wander the websphere looking at options. I've learned some things in the search, but I always looked back at Zope, even tho' my wife liked Enhydra's logo, she thinks the otter is cute. Zope's current logo is much nicer than the Purple and Gold. :)
If ZEO currently scales well, then DC needs some testimonials on their website.
Right.
There definitely needs to be some documentation best ways to... As you said, there is a consistent recurring thread of is ZServer alone or ZServer behind Apache better. Should I use a RDBMS or ZODB.
Right.
Some of us have no legacy issues. From scratch what's the best way to design...
Ditto.
I agree the solving of Performance/Scalability and Documentation issues would definitely help convert lookers into believers.
That's a good way to phrase it.
Amen!
--Paul
Jimmie Houchin