Greetings Zopesters, I'm about to prepare a content system that is interface by HTML and/or shockwave. A list of coordinates is posted and dated records are read in from (the decision here) the DBMS or ZopeFolderZClass scheme plotted to these coords. It happens quite often and it is a multi-user system, to optimally executing multiplie simultaneous requests are the largest deciding factor. What I am wondering is if to build my content hiearchy using Folders with Attributes on a ZClass schema, or Postgres records with BLOBS contianing the HTML. What is the performance factor? A) Zope is single threaded, but capable of decoupling, thus a multi-threaded dbms will open many threads per user. So 5 users hit the site, 5 threads are started and Zope will RESPOND back single-threadedly. Yes? (1 point for postgres) B) Folder reading doesnt require anymore than Zope, thus there is less activity as far as dbms threading, so folder reading in per request is quicker, and more dependent of the servers I/O speed (U2W scsi, RAID, whooaa!) and the content creation model is easier to tackle to create objects. (1 point for Folders/Attributes/ZClasses) These are the two biggest pros and cons but I hope someone like Dieter and Chris that knows the guts of zope can advise me. Perhaps a dual-usage schema could be used. Objects are created using ZClasses and reside in Zope (for crawler indexing and client bookmarking, and no query strings) and each has its index duplicated in Zope for application indexing purposes. The actual Zope object wouldnt be requested until its page was needed, and not its presence in an index. I think that would be the best of both worlds. Comments, or did I just solve my own problem? Thanks as always. Paz
Hi Zopers, That's generally an interesting discussions for all Zope applications which are somewhere between low- and high-volume data amount. I suppose that simple, low-volume data is best handled by some Python objects stored in ZODB. But where are the limits of internal features such as ZCatalog, for instance? Do they keep up with indexing tools such as mg?
Perhaps a dual-usage schema could be used. Objects are created using ZClasses and reside in Zope (for crawler indexing and client bookmarking, and no query strings) and each has its index duplicated in Zope for application indexing purposes. The actual Zope object wouldnt be requested until its page was needed, and not its presence in an index. I think that would be the best of both worlds. Comments, or did I just solve my own problem?
I like the approach, thought about something like that too -- kind of a proxy in Zope and the real data in some external database. I'd appreciate if someone could point me (and others) to some docs or howto's explaining more details on this problem! Lars
Lars, Cant say about How-Tos on the subject otherwise i wouldnt have asked ; ) , but a discussion could be started on how a product could be created that did this. I've for a while envisioned a product that would create a ZClass through variables. You are supplied with a management screen to create SQL tables by using drop down lists and a 'number of fields' field. This product would then create the SQL database and simple ZSQL methods for retrieving and adding. (I once made a clunky but powerful generic dtml method that checks security and thus not needing to custom make ZSQL methods).
From this discussion I see another option could be that this product creates a SQL index specificaly for indexing Zope objects that contain primary/foreign key relations to the database. This allows SQL to be used for content indexing (CPU intensive, but not client download intensive) actions, and allow Zope to store the actual HTML. Any further use would have to be customizable, but at least a datamodel creation tool would quickly help alot of zopers.
I find that using SQL exclusivley is more work than the performance gain, and content in SQL is even worse because you cannot index or search or crawl it. Paz -----Original Message----- From: Lars von Wedel [mailto:vonWedel@lfpt.rwth-aachen.de] Sent: Thursday, July 12, 2001 12:01 PM To: Paul Zwarts Cc: Zope Subject: Re: [Zope] performance Hi Zopers, That's generally an interesting discussions for all Zope applications which are somewhere between low- and high-volume data amount. I suppose that simple, low-volume data is best handled by some Python objects stored in ZODB. But where are the limits of internal features such as ZCatalog, for instance? Do they keep up with indexing tools such as mg?
Perhaps a dual-usage schema could be used. Objects are created using ZClasses and reside in Zope (for crawler indexing and client bookmarking, and no query strings) and each has its index duplicated in Zope for application indexing purposes. The actual Zope object wouldnt be requested until its page was needed, and not its presence in an index. I think that would be the best of both worlds. Comments, or did I just solve my own problem?
I like the approach, thought about something like that too -- kind of a proxy in Zope and the real data in some external database. I'd appreciate if someone could point me (and others) to some docs or howto's explaining more details on this problem! Lars
----- Original Message ----- From: "Lars von Wedel" <vonWedel@lfpt.rwth-aachen.de> To: "Paul Zwarts" <paz@www2.oratrix.nl> Cc: "Zope" <zope@zope.org> Sent: Donnerstag, 12. Juli 2001 05:00 Subject: Re: [Zope] performance
But where are the limits of internal features such as ZCatalog, for instance? Do they keep up with indexing tools such as mg?
What is mg ? ============================================================================ ==== Andreas Jung andreas@digicool.com Digital Creations "Zope Dealers" http://www.zope.org
Hi, mg is the indexing system described in the book 'Managing Gigabytes' by Witten et al. They claim it to be very fast and scalable. You can find some information on the book as well as the source of mg at: http://www.cs.mu.oz.au/mg/ As this is a product dedicated to compression and indexing of huge amounts of data I am just curious whether ZCatalog keeps with this or if ZCatalog was designed to do 'simple searches' within a rather small website. Lars Andreas Jung wrote:
----- Original Message ----- From: "Lars von Wedel" <vonWedel@lfpt.rwth-aachen.de> To: "Paul Zwarts" <paz@www2.oratrix.nl> Cc: "Zope" <zope@zope.org> Sent: Donnerstag, 12. Juli 2001 05:00 Subject: Re: [Zope] performance
But where are the limits of internal features such as ZCatalog, for instance? Do they keep up with indexing tools such as mg?
What is mg ?
============================================================================ ==== Andreas Jung andreas@digicool.com Digital Creations "Zope Dealers" http://www.zope.org
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Hi: Just a quick heads up so that you all are aware of a possible Zope competitor or, if you take a opposing view, to be aware of someone blatantly copying Zope ;-) Macromedia has launched its SiteSpring product which allows you to collaborate, manage and communicate when doing web projects. The site is at http://www.macromedia.com/software/sitespring/ and, if you watch the Quicktime movie or install the demo, even the goddam menus are Zope-alike. Emulation aside, it does raise some issues, namely on the workflow/communicate functionalities. How is Zope going to cope ? Do we have products available that bring Zope up to the same level ? C U! -- Mario Valente
Hi Paz, You may want to look at ZPatterns for this. It allows you to mix/match ZODB/RDBM in a most flexible way. It also has nice 'RuleAgents' that allow you to perform arbitrary actions/'triggers' at object create/change/delete events. -steve Paul Zwarts wrote:
Perhaps a dual-usage schema could be used. Objects are created using ZClasses and reside in Zope (for crawler indexing and client bookmarking, and no query strings) and each has its index duplicated in Zope for application indexing purposes. The actual Zope object wouldnt be requested until its page was needed, and not its presence in an index. I think that would be the best of both worlds. Comments, or did I just solve my own problem?
Thanks as always. Paz
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Paul Zwarts wrote:
Greetings Zopesters,
I'm about to prepare a content system that is interface by HTML and/or shockwave. A list of coordinates is posted and dated records are read in from (the decision here) the DBMS or ZopeFolderZClass scheme plotted to these coords. It happens quite often and it is a multi-user system, to optimally executing multiplie simultaneous requests are the largest deciding factor.
What I am wondering is if to build my content hiearchy using Folders with Attributes on a ZClass schema, or Postgres records with BLOBS contianing the HTML. What is the performance factor?
BLOB support in Postgres is rather weak. Zope on the other-hand excels in this area for small to medium sized files.
A) Zope is single threaded, but capable of decoupling, thus a multi-threaded dbms will open many threads per user. So 5 users hit the site, 5 threads are started and Zope will RESPOND back single-threadedly. Yes? (1 point for postgres)
Zope is multi-threaded and defaults to 4 active threads, which can be changed.
B) Folder reading doesnt require anymore than Zope, thus there is less activity as far as dbms threading, so folder reading in per request is quicker, and more dependent of the servers I/O speed (U2W scsi, RAID, whooaa!) and the content creation model is easier to tackle to create objects. (1 point for Folders/Attributes/ZClasses)
Zope excels at read-intensive operations, much less-so at writing. If performance is an issue, I would recommend not using ZClasses and instead developing your objects as products directly in Python.
These are the two biggest pros and cons but I hope someone like Dieter and Chris that knows the guts of zope can advise me.
It really depends on 4 main factors in my mind, whether to use Zope or a RDBMS: 1. What is the nature of the data itself? Zope by itself is far more efficient dealing with BLOBS than Zope+Postgres would be. Your implementation would be much cleaner too. 2. How do you need to query the data? Postgres has the advantage of SQL here, Zope the text-indexing powers of ZCatalog and the straightforward ZODB. 3. What is the read/write ratio. Postgres scales much better for write-intensive operations. 4. What is the simplest way to implement your solution without great compromise? If you don't have a strong reason to use an RDBMS, then don't.
Perhaps a dual-usage schema could be used. Objects are created using ZClasses and reside in Zope (for crawler indexing and client bookmarking, and no query strings) and each has its index duplicated in Zope for application indexing purposes. The actual Zope object wouldnt be requested until its page was needed, and not its presence in an index. I think that would be the best of both worlds. Comments, or did I just solve my own problem?
Again, try to find the simplest solution to your problem, but no simpler. And remember, the most reliable, best performing and easiest parts of your system to maintain are the parts that aren't there.
Thanks as always. Paz
Good luck! -- | Casey Duncan | Kaivo, Inc. | cduncan@kaivo.com `------------------>
participants (6)
-
Andreas Jung -
Casey Duncan -
Lars von Wedel -
Mario Valente -
Paul Zwarts -
Steve Spicklemire