On 2/14/07, Philipp von Weitershausen <philipp@weitershausen.de> wrote:

There will definitely be a Zope 2.11 and at this point I see no reason why there shouldn't be a Zope 2.12. We (the Zope Community, not necessarily Zope Corporation) will maintain Zope 2 as long as it's necessary. "Maintaining" in this case also means integrating more Zope 3 technologies with each release. Nowadays (Zope 2.10) you can already write applications, or at least base libraries, that work on both Zope 3 and Zope 2.

This is very helpful for setting a planning horizon, thank you.

I'm surprised you're experiencing poor scaling. I think Zope can scale pretty well, or at least it can be *made* to scale pretty well. It all depends on your setup and, of course, on your application.

I can't swear to the accuracy of what I'm about to say but it is an operating hypothesis that describes accurately behavior that we are seeing. Our application is highly dynamic, by which I mean that satisfying any given request may require a great deal of database activity. The database in question is almost always Oracle; I'm not counting the ZODB accesses, here. Oracle will handle a great many concurrent requests fairly gracefully, but its average transaction time is not always excellent, especially for moderately complicated queries. While most queries are sub-second, populating a page that requires multiple queries to fill in all its data fields, or even populating a page with only one or two complicated queries, can require greater than a single second of database I/O. This cost is offset by the ability to have other threads processing while your first thread waits for the database to come back. Zope likes to have 4 threads processing requests. This preference is documented in several places, and experiments have historically borne out the "4 thread hypothesis". We have been unable to realize significant performance improvements by increasing the number of available worker threads, but we are able to realize performance improvements by pointing two Zope servers against the same database. Zope isn't using much CPU, and it's not using much memory, but it peaks out at 4 concurrent requests; everything else goes in a queue behind them. If those 4 concurrent requests happen to take 4 or 5 seconds each, the queue builds deep behind them, possibly with very simple one-offs. Historically, we've addressed this by sticking another Zope out there. We have plans to run multiple Zope instances on a single machine, like they do (did?) at plone.org; this still requires configuration management work and testing. But the feeling is that we have a lot more hardware than we really should need, to handle the volumes we're dealing with, and we have that hardware because we need more "channels" to handle our concurrent traffic load.

As for source control, I figure all of your code (DTML, yuck) is in the ZODB. This went out of fashion a long time ago, most serious development happens on the filesystem (in Python packages) which can obviously be source-controlled very well.

A good deal of the code is in the ZODB, because it's (yuck) DTML documents, yes. We have another sizable chunk that lives on the filesystem. The challenge in managing the build/deploy process has been trying to find a way to keep those two conjoined. I suspect that any dependency on the ZODB at all is likely to be considered an impediment, and my quick eyeballing says it's not gone or even really optional under Zope 3 (but I'm sure we could work something out.)

Regarding "oh you'll hafta start over", it's pretty much true, if you want to switch to Zope 3. But nobody says you have to. You can do it incrementally by porting some of your app's functionality to Zope 3 components step by step (as mentioned already, most work in Zope 2). Big projects like Plone aren't rewriting their whole codebase either, but new development is completely Zope3-based. Their target platform still is Zope 2, though.

"You'll hafta start over" is only ever true in degrees, though, and I'm still trying to figure out what the degrees are. The business logic will still be the same. Can we hack together a DTML processor that allows us to export the DTML documents to the filesystem and publish them from there? Maybe, I don't know. Did someone else already do that? Don't know that either. How dramatically will our products need to change? Probably 75% of our Python code is written in a bastardized form of ExternalMethod; we might be able to leverage that unfortunate architectural choice to significant advantage during our porting phase. These are the kinds of questions I have, and I think the answers probably aren't easy or someone would have offered them up. Someone's going to need to learn enough Zope 3 to answer the questions. I'm not sure it will be me, but maybe. :) Thanks for all the insight, I'm making plans and pleas and I have a direction for my research now. :) Thanks, --G.

On Thu, Feb 15, 2007 at 11:50:38AM +0100, Philipp von Weitershausen wrote:

What's already possible is to have a minimal ZODB with only one persistent object: a SQLObject or SQLAlchemy container. That's a container (e.g. like a folder) whose items aren't persisted in the ZODB but come from a relational database.

This works fine in a Zope 2 / Five world too. We have apps that have nothing in the ZODB but a Z*DA instance. Then we have zope 3 views and adapters that acquire that instance and use it to talk to the database, using ZSQL to build the queries. We haven't drunk the ORM cool-aid yet. :) In other words: The ZODB may be there but you can pretty much ignore it if you don't need it :)

You seem to already have come to the conclusion that having code live in both the filesystem and the ZODB can be painful. I think a good first step for you would be to migrate your remaining ZODB-based code to the filesystem. That not only makes deployment easier, you're also free to refactor it then (e.g. using Zope 3 idioms). In the long run, this also means saying good-bye to things like External Methods because they require code on the filesystem and configuration in the ZODB.

+1 Here's my off-the-cuff attempt at "How To Safely Move A Big Legacy App Out of the ZODB". First, get and install FSDump (google for it). Add a Dumper to a folder containing DTML (it works recursively on child folders too), configure the Dumper, click the "Save and dump" button, and voila, there's your code on the filesystem. It'll store properties too, in files named like *.metadata. Then you need a way to use the stuff from the filesystem. FileSystemSite would be ideal, because it understands those .metadata files: http://www.infrae.com/download/FileSystemSite Using it is a bit non-obvious if you're not familiar with CMF, from which it was extracted. You'll need to create a minimal Product on the filesystem that consists of: - a directory in Products, let's call it Products/mystuff - an __init__.py at Products/mystuff/__init__.py that looks like this: from Products.FileSystemSite.DirectoryView import registerDirectory registerDirectory('mysubdir, globals()) - a subdirectory at Products/mystuff/mysubdir. Put the files and directories you dumped in here. That's it. Restart Zope, add an instance of Filesystem Directory View, and you'll be prompted for the directory to choose; the only choice will be the "mysubdir" you registered above. Click OK and you're done. Check this new Product into source control and rejoice. One gotcha - files ending in .dtml will be treated as DTML Methods, not DTML Documents. So if you relied on the semantic difference between those, you will have some issues to sort out. Another gotcha - you'll need another plan to deal with those External Methods, since AFAIK neither Dumper nor FileSystemSite will handle them. One expedient if slightly tedious technique would be to just move the External Method files out of Extensions/ and into $INSTANCE_HOME/lib/python/. Then use calls to allow_module() so they can be imported by through-the-web code (google will tell you more about that). Then, delete each external method from the ZMI and in its place add a Script (Python) with the same id. Such a script would have a body something like: from my_ext_method_module import myExternalMethodFunction return myExternalMethodFunction(arguments) Once that's done, those scripts can be Dumped into your filesystem code, and all the DTML that includes calls to them should work. A final gotcha - if you have ZClasses, I can't help you :) At this point, you have the same application you started with, it's just on the filesystem. Not as clean as a rewrite, but a hell of a lot more expedient - faster and less risky. Next you'll want to start adding regression tests to your product (ZopeTestCase and Zelenium might be useful tools). Then you have a safety net so you can confidently start refactoring as needed. *This* is when you can start thinking about rewriting stuff :-) I'd put this email somewhere on zopewiki.org, but it seems to be down at the moment (Bad Gateway).

It is certainly possible to execute filesystem-based DTML (using Globals.DTMLFile).

Right. But that's a bit of a pain when you have dozens or hundreds of DTML methods and your first priority is just to get stuff out of the ZODB and into source control without breaking anything. -- Paul Winkler http://www.slinkp.com