Some tough questions about zope... (long)
Hi all, This is quite a long email where I ask a lot of questions about Zope, but I hope you'll be lenient with me as I think I have a few hard decisions to make! Some background: I'm one of the web developers here, with a remit to implement a curriculum database that can; (a) support the Medical Faculty here (b) get rolled out to three other Medical-related sites, each with their own 'look and feel' (c) possibly implement a University-wide curriculum database. The curriculum database is essentially delivering online study guides (over 100 for each Faculty) whose source is originally from very well marked up Word Documents. In addition, we include multiple choice questions, file upload of lecture notes etc. and some 'active' web pages showing dynamical processes in medicine etc. The curriculum database needs to be searchable across its entire breadth (although initially restricted to the one site), so that students and faculty can find occurences of, say, 'lung cancer' in lecture resources - this is to plan for curriculum management, so that we can find out if we're pushing subjects too much (or indeed, whether we're pushing a subject at all!). We have other applications (nearly all of which are written in Perl/PHP3) and want to seemlessly integrate them into the curriculum database. Currently we do all our development work on Unix, using Apache virtual web servers, MySQL as our database engine and Perl running as a CGI for most of our applications. My input has been to introduce PHP-3 (an Apache module with a bulit-in XML parser) and some XML knowledge to the group. I've been looking at a scenario where our study guides are converted in XML using Omnimark and held as static XML documents in a database. They are then rendered into HTML/RTF and put into static web directories. The student can run a program that extracts elements from selected XML documents and creates a customised set of HTML pages or RTF document that is built for them 'on the fly'. My manager was a bit worried about all these static pages lying about and to be honest so am I, but extracting the relevant XML from a database to render into individual HTML pages seems a massive drain on the system. I found out about Zope Monday morning, dl'ed it, installed it, got one or two things working... and have been worried sick ever since. My problem? is that I can see how Zope can do so much for us, but it seems to need a hell of a gut-wrench to do so. There also doesn't seem to be an easy 'half-way' house to move to Zope. I've read a lot of the documentation, and am doing so constantly, but the wisdom of the list would be very helpful indeed. The questions: a) Is there a way for us to 'dump' the non-dynamic parts of a Zope database into static HTML pages. This would act as a way to say to the team "look, we have a parachute!". b) Can I integrate with the Perl/PHP3 programs that we have (the Perl programs are mainly .cgi, but the PHP3 programs are server-side scripted, and as such are embedded into the HTML itself). This is so that I can c) Is there any way of using the XML documents that I'm creating into the database and parsing them to produce either HTML or RTF output? d) I know that this a terrible question to ask... but is Zope likely to 'remain' open-source (we're worried about getting stuck with a proprietary system)?. e) Can I 'point' an Apache virtual server to run a Zope cgi (ie dropping the :port requirment). f) I'd like to have the option to have user authentication information coming from a database - is this possible? g) We have something like 200 main level directories holding the study guides, each with up to 200 files in them - is that pushing the Zope database too far? h) We also need resources such as QuickTime movies, images PowerPoint slides etc. to be uploaded - would they sit in the database or can they be held somewhere else (and what happens about file name conflicts)? i) Does Zope have the equivalent of 'server logs', which we use for research. j) Is it possible to have a 'pointer' at a location in the Zope database that will point to a 'real, honest to goodness' HTML file on the server (ie something like a redirect..oops I think I may have answered my own question) and lastly k) What kind of loads have people put Zope under (10,000 +hits a day, 100k?). We have one application which has 170 Medical Students sitting in front of screens and clicking a button at the same time (they're selecting their final year options) which then starts 170 CGI-based Perl processes off interrogating and updating the database. This really grinds the system down... I hope that I haven't bored everyone to tears here - it's just that Zope seems to do an awful lot of what I need, and I'd really like to know what I'm letting myself in for if I start using it for everything I do.... many many thanks, tone. ------ Dr Tony McDonald, FMCC, Networked Learning Environments Project The Medical School, Newcastle University Tel: +44 191 222 5888 Fingerprint: 3450 876D FA41 B926 D3DD F8C3 F2D0 C3B9 8B38 18A2
Let me try to answer (some) of your questions:
The questions: a) Is there a way for us to 'dump' the non-dynamic parts of a Zope database into static HTML pages. This would act as a way to say to the team "look, we have a parachute!".
I will be looking into this, as soon as I have the time. The answer for now is: probably, because everyone wants to download a local copy of part of any website at some point. There are tools out there to do this, I haven't looked for them yet.
b) Can I integrate with the Perl/PHP3 programs that we have (the Perl programs are mainly .cgi, but the PHP3 programs are server-side scripted, and as such are embedded into the HTML itself). This is so that I can
Just run Zope via Apache (see below).
c) Is there any way of using the XML documents that I'm creating into the database and parsing them to produce either HTML or RTF output?
Python is very capable of parsing XML. You will have to write an External Method to handle this, though. An example of XML parsing can be found at http://www.zope.org/Download/Contrib/XMLConfig.tar.gz. This Zope Product parses a static XML file for use in Zope.
d) I know that this a terrible question to ask... but is Zope likely to 'remain' open-source (we're worried about getting stuck with a proprietary system)?.
I certainly do hope so. I intent to use it en expand on it heavily anyway, and sodo others. These extensions will generally be Open Source themselves, making it hard for Digital Creations to pull Zope out of the Open Source domain.
e) Can I 'point' an Apache virtual server to run a Zope cgi (ie dropping the :port requirment).
Yes. The doc/WEBSERVER.txt will start you on this. Be warned though, there are some problems with Apache and Zope integration, for reasons that others are better equiped to explain.
f) I'd like to have the option to have user authentication information coming from a database - is this possible?
Yes, you can find the UserDB Product at http://www.zope.org/Download/Prereleases/UserDB/UserDb-1.2.0-nonbin.tar.gz.
g) We have something like 200 main level directories holding the study guides, each with up to 200 files in them - is that pushing the Zope database too far?
Nah, Zope has handled much more than that allready. Ask Paul =).
h) We also need resources such as QuickTime movies, images PowerPoint slides etc. to be uploaded - would they sit in the database or can they be held somewhere else (and what happens about file name conflicts)?
Zope can store any filetype you want it to. Filename conflicts are handled the same way a filesystem handles them: In the same Folder, you cannot have two objects with the same name at once. But because it is a hierachical system, you can use the same name in two different folders.
i) Does Zope have the equivalent of 'server logs', which we use for research.
ZopeHTTPServer's output is it's serverlog. You can start it up with STDOUT redirected into a file. When you use Apache as your webserver, all requests will be logged through Apache itself.
j) Is it possible to have a 'pointer' at a location in the Zope database that will point to a 'real, honest to goodness' HTML file on the server (ie something like a redirect..oops I think I may have answered my own question)
You said it yourself =).
and lastly k) What kind of loads have people put Zope under (10,000 +hits a day, 100k?). We have one application which has 170 Medical Students sitting in front of screens and clicking a button at the same time (they're selecting their final year options) which then starts 170 CGI-based Perl processes off interrogating and updating the database. This really grinds the system down...
Zope will be much much faster. I can't give you any direct numbers, maybe someone else on this list can. You basicly don't have the overhead of starting a fullblown new process for every request, and Zope pools Database connectiosn for you, so you don't have the initial connection overhead either. And, your code isn't compiled everytime, like with Perl. -- M.J. Pieters, Web Developer | Antraciet http://www.antraciet.nl | Tel: +31-35-6254545 Fax: +31-35-6254555 | mailto:mj@antraciet.nl http://www.antraciet.nl/~mj | PGP: http://wwwkeys.nl.pgp.net:11371/pks/lookup?op=get&search=0xA8A32149 ------------------------------------------
On Wed, Mar 03, 1999 at 02:41:53PM +0000, Tony McDonald wrote: ,----- | Hi all, | This is quite a long email where I ask a lot of questions about Zope, | but I hope you'll be lenient with me as I think I have a few hard | decisions to make! I know the feeling. I was choosing between ColdFusion and Zope for developing my new application and had some similar questions. Ultimately, I chose Zope. [nice application description deleted] | a) Is there a way for us to 'dump' the non-dynamic parts of a Zope | database into static HTML pages. This would act as a way to say to | the team "look, we have a parachute!". Right now, there isn't. This question has come up repeatedly on the list, to the point where DC is getting sick of hearing it and thinking of actually implementing something :) They have said it would be pretty easy, and I believe it. I think the basic idea is to write a method that will go through the Folders and use the DocumentTemplate code to render out the pages. | b) Can I integrate with the Perl/PHP3 programs that we have (the Perl | programs are mainly .cgi, but the PHP3 programs are server-side | scripted, and as such are embedded into the HTML itself). This is so | that I can If you're using Zope with Apache, you use mod_rewrite to get the authentication headers passed to Zope. You can also use mod_rewrite to decide how URLs map to content on your server. For example, you can have all of the content on your site go through your Zope.cgi, but /cgi-bin/* can go directly to your CGI or PHP files. You can access MySQL from Zope, PHP and Perl, so they can all use the same data source for the databases. | c) Is there any way of using the XML documents that I'm creating into | the database and parsing them to produce either HTML or RTF output? I'll leave this one to people more knowledgeable about XML. | d) I know that this a terrible question to ask... but is Zope likely | to 'remain' open-source (we're worried about getting stuck with a | proprietary system)?. I think it would be tough for DC to try to take it back at this point :) Even if DC did decide to not release any more updates to Zope, the community would probably continue working from the 1.10.2 code. Anyhow, DC had sound business reasons to release Zope as Open Source. Over time, they will likely get a lot of benefits by having done so and by encouraging outsiders (like us) to work with and improve Zope. | e) Can I 'point' an Apache virtual server to run a Zope cgi (ie | dropping the :port requirment). Yes. This uses a technique called Persistent CGI. It works quite well. There is also a hope that there will be better integration with Apache in the future. Also, as ZServer stabilizes, you may find good reason to migrate from Apache to it. | f) I'd like to have the option to have user authentication | information coming from a database - is this possible? There is an unsupported bit of code called UserDB which is designed for this. | g) We have something like 200 main level directories holding the | study guides, each with up to 200 files in them - is that pushing the | Zope database too far? Though I have no first hand experience with a Zope DB this large, the DC guys have said that they've worked with some very large DBs. | h) We also need resources such as QuickTime movies, images PowerPoint | slides etc. to be uploaded - would they sit in the database or can | they be held somewhere else (and what happens about file name | conflicts)? You could put them in either place. There would probably be some advantages to putting them in the database. For example, your standard Zope security rules will apply to the downloading and updating of the files. | i) Does Zope have the equivalent of 'server logs', which we use for research. I don't know about ZServer, but if you use Apache as your web server you will be able to use your standard server logs. | j) Is it possible to have a 'pointer' at a location in the Zope | database that will point to a 'real, honest to goodness' HTML file on | the server (ie something like a redirect..oops I think I may have | answered my own question) Yes, you can do a redirect. | and lastly | k) What kind of loads have people put Zope under (10,000 +hits a day, | 100k?). We have one application which has 170 Medical Students | sitting in front of screens and clicking a button at the same time | (they're selecting their final year options) which then starts 170 | CGI-based Perl processes off interrogating and updating the database. | This really grinds the system down... I've heard it said that Zope should be able to handle 500k hits or more on reasonable hardware. Of course, this depends on exactly what the server has to go through for each of those hits, etc. DC is also working on concurrency for Zope 2.0. This will probably enable you to handle a whole bunch of requests. `----- Good luck! Kevin -- Kevin Dangoor kid@ans.net / 734-214-7349
participants (3)
-
Kevin Dangoor -
Martijn Pieters -
Tony McDonald