RE: [Zope] Zope needs this (and Dynamo has it)
And here follows part two of my follow-up. (I'm all thumbs today -- why is Alt+S so damn accessible on a PC keyboard? ;-)
From: Thomas Stenhaug [mailto:thomas@src.no] Sent: Sunday, March 05, 2000 2:34 PM To: Alexander Staubo Cc: Zope Mailing List (E-mail) Subject: Re: [Zope] Zope needs this (and Dynamo has it)
| This situation has nothing to do with open vs. closed, either. Just | like Zope we're talking about proprietary (as opposed to standarized | -- DTML may be open, but it's still proprietary)
I don't quite grok what "the situation" is referring to,
Read the preceding paragraph, about Java and the "cost of training a person in the arts of Zope".
but I think you're wrong here, or at least mistaken regarding the meaning of "proprietary". Proprietary, according to Merriam-Webster, means this:
[cut from Merriam-Webster] Function: adjective Etymology: Late Latin proprietarius, from Latin proprietas property -- more at PROPERTY Date: 1589 1 : of, relating to, or characteristic of a proprietor <proprietary rights> 2 : used, made, or marketed by one having the exclusive legal right <a proprietary process> 3 : privately owned and managed and run as a profit-making organization <a proprietary clinic> [/]
I mean proprietary as being controlled by a private party -- owned, in a very liberal sense -- as opposed to being standarized. XML is not proprietary. DTML and Java are. I'm not as knowledge about IP laws as I would like to be, so I can't really defend my use of this word or whether DTML is an intellectual property of Digital Creations. Perhaps somebody else more suited to the task could clarify.
[...] | and Zope, despite its many powerful features, is not big enough for | truly big things. | | This is a two-part problem. The first part is that Zope's bullshit | factor is too low; [...] | We tech people know better, we know a shiny surface tells us nothing | about the engine underneath, but we tech people are not holding the | crucial component -- those big brown bags of money.
I'm not sure if I agree it's a problem, then, that Zope lacks in the big-department. :-)
If the suits in managements (assuming you work in such a place) tell you to go learn Dynamo because that's what they want to use, wouldn't you say it's a problem?
| [under Zope's surface] lies a powerful engine with scalability | problems.
What part of Zope's engine are you referring to here? Do you mean Python in general, are some specific part? Like ZODB?
I definitely believe that Python is Zope's largest problem right now. It's also, from a technical point of view, one of it's biggest assets -- Zope is becoming the killer application of Python, and might eventually become the driving force to evolve Python, which is, I believe, suffering from draught into the contributor department. Skilled developers aren't flocking to Python to make it faster. Not right now, anyway. My hope is that Zope will change this, because Python has possibly one of the slowest interpreters on earth. Python zealots -- and there is such a beast -- will tell you that Python's performance is less of a matter because what you sacrifice in speed you gain in power, ie. computational expressiveness through a high-level, dynamic language. For an example of a good, Python-like, dynamic language that is also very fast, take a look at Dylan. In current implementations, Dylan is not interpreted but rather compiled to native code, but in theory there is nothing stopping anyone from writing a highly efficient Dylan interpreter. Dylan's magic comes from its use of type hints, a system which I had recognized as likely pivotal to improving Python's interpreter long before I knew of Dylan. In Dylan, specifying a variable's type is optional. When omitted, the Dylan compiler will usually be able to infer the type from what you do with the variable (ie., it sees you're assigning an integer to it; hence it can assume that, for at least a stretch of execution, the variable remains an integer). When you explicitly declare a variable's type, Dylan applies the necessary code optimizations (such as using tight integer maths or string copying) as well as static type checking. In Python, variables can hold virtually anything, and this incurs a huge amount of checks. Just imagine -- while Zope is running, thousands and thousands of the same little checks are performed continuously, over and over again, amounting to no uncertain overhead. Python's biggest overhead comes from its dynamism -- the dynamic typing, as explained, combined the object allocation system: most things is an object, so a lot of objects are constantly spawned and torn down, often in rapid succession because the compiler/interpreter can make few hard analyses of when stuff is needed. Python also has a high function-call overhead.
| So far the only way to scale properly with Zope is a combination of | a load-balancing clustering system (eg., TurboLinux' TurboCluster | Server, which I can personally vouch for -- brilliant stuff -- or | Linux Virtual Server) with Digital Creations' ZEO, of which I've | only read about.
I actually thought of ZEO as a very nice way to scale Zope.
I agree.
| There are Zope sites out there that are suffering scalability | problems (I won't mention specifics, other than that the project I'm | involved in is one of them).
It would be very much appreciated if you could share some specifics.
No. Ask around, do some research. Ask Digital Creations -- they probably know of a few, although I'm guessing that they might feel it would hurt their reputation to tell. Many companies using Zope might not want to tell you because it would hurt their position, especially if they're public companies. Here's one specific case that I'm personally struggling with. I have a page that returns a large number of dynamic database results in a relatively simple table. The results are first munged by a small Python script. This stuff is slow. Internally, as far as I can determine (through shallow study of the database code), Zope allocates a row object for every row you access -- even if the row has been returned earlier on. Zope has a 1-row cache for cases when you repeatedly ask for the same row in succession. Zope also has a row cache that is useful for static result sets. Yet, for dynamic result sets, the processing overhead is huge. The overhead isn't so higher than any other overhead in other areas in Zope, but it's especially painful here, because it affects the generation time of a single page, whereas other kinds of overhead are more evenly distributed across multiple pages and multiple users.
[...] | As far as pollution goes, assuming Zope exports the appropriate data | -- in whatever form -- then support for | performance/monitoring/management APIs such as SNMP, PCP, WBEM, and | to a lesser extent PDH, aren't difficult to isolate into add-on | products.
You are right about that, and I think that's a better way to provide such support, rather then building specific support for any of them into Zope.
Exactly. However, the necessary counters must be built into Zope at some point, and exposed through a common (XML? Python/XML-RPC?) API. Then we could attach publishers for SNMP, HTML, XML, whatever. Which reminds me. Does Zope have a mechanism for extending the Control_Panel namespace other than through the Product sub-folder? I'm too lazy to check, and I don't remember if this was ever discussed. -- Alexander Staubo http://alex.mop.no/ "`Ford, you're turning into a penguin. Stop it.'" --Douglas Adams, _The Hitchhiker's Guide to the Galaxy_
Alexander Staubo wrote:
I definitely believe that Python is Zope's largest problem right now. It's also, from a technical point of view, one of it's biggest assets -- Zope is becoming the killer application of Python, and might eventually become the driving force to evolve Python, which is, I believe, suffering from draught into the contributor department. Skilled developers aren't flocking to Python to make it faster. Not right now, anyway.
Hm, skilled developers do hack on Python! There's the Python people at CRNI, there's Christian Tismer in Germany doing Stackless Python (which offers interesting arcane features and a mild performance improvements), there's the whole JPython group, there's John Skaller with his Vyper project (a Python interpreter/compiler in Ocaml, one of the reasons is more speed). Then there's the Python Types-SIG, which I urge you to look at if you haven't already.
My hope is that Zope will change this, because Python has possibly one of the slowest interpreters on earth. Python zealots -- and there is such a beast -- will tell you that Python's performance is less of a matter because what you sacrifice in speed you gain in power, ie. computational expressiveness through a high-level, dynamic language.
It is _less_ of a problem, but most Pythoneers are indeed interested in speeding of the language. I haven't heard of zealots who want to hold back speedups, as long as it doesn't do away with the power.
For an example of a good, Python-like, dynamic language that is also very fast, take a look at Dylan. In current implementations, Dylan is not interpreted but rather compiled to native code, but in theory there is nothing stopping anyone from writing a highly efficient Dylan interpreter. Dylan's magic comes from its use of type hints, a system which I had recognized as likely pivotal to improving Python's interpreter long before I knew of Dylan.
It'll be pretty difficult to make a beast like Zope exploit type-hints, though, I think. By the way, the Types-SIG is working on type extensions to Python, partially for performance (OPT), partially for better error checking (ERR) and partially for better documentation (DOC). See: http://www.python.org/sigs/types-sig/ And also see the new compiler SIG: http://www.python.org/sigs/compiler-sig/
In Dylan, specifying a variable's type is optional. When omitted, the Dylan compiler will usually be able to infer the type from what you do with the variable (ie., it sees you're assigning an integer to it; hence it can assume that, for at least a stretch of execution, the variable remains an integer). When you explicitly declare a variable's type, Dylan applies the necessary code optimizations (such as using tight integer maths or string copying) as well as static type checking. In Python, variables can hold virtually anything, and this incurs a huge amount of checks.
Type inference is not a simple thing to add to the language, though..
Just imagine -- while Zope is running, thousands and thousands of the same little checks are performed continuously, over and over again, amounting to no uncertain overhead.
That's of course true. I heard Christian Tismer talk about making a register machine interpreter for Python, which could potentially speed things up quite a bit. I shall ask him about this idea again sometime. :)
Python's biggest overhead comes from its dynamism -- the dynamic typing, as explained, combined the object allocation system: most things is an object, so a lot of objects are constantly spawned and torn down, often in rapid succession because the compiler/interpreter can make few hard analyses of when stuff is needed. Python also has a high function-call overhead.
All true. Anyway, the only thing you're wrong about is that Pythoneers aren't interested in improving performance. They are, but I grant you they're not performance nuts like users of C or C++. :) Regards, Martijn
participants (2)
-
Alexander Staubo -
Martijn Faassen