Zope's database capability
I am in the process of evaluating Zope and so far it looks great. I have one question about Zope's database: How does it compare in performance to a relational database server? I am think about a scenario such as a recruitment site with a large database of jobs and candidates. Would the Zope database manage it OK or is it normal to run a database server such as mySQL alongside it for the heavy-duty stuff? Is it possible to build complex queries on the Zope database? Has anyone had any experience of building a big database-driven site in Zope, and if so did you build it all using the Zope database? OK, so that turned into more than one question, but any comments would be gratefully received (I am hoping to be persuaded that Zope is great) Cheers Tom.
[Tom Nixon]
I am in the process of evaluating Zope and so far it looks great.
I have one question about Zope's database: How does it compare in performance to a relational database server? I am think about a scenario such as a recruitment site with a large database of jobs and candidates. Would the Zope database manage it OK or is it normal to run a database server such as mySQL alongside it for the heavy-duty stuff?
Yes, it is feasible and normal to use a relational database with Zope. Zope gives you very good support for working with relational databases. You can use ODBC or various database-specific database drivers. Zope will manage transactions for you.
Is it possible to build complex queries on the Zope database?
The Zope database doesn't have an extensive query language comparable to SQL, and there is no native join-like construction as best I understand things. Cheers, Tom P
On Thu, Jan 24, 2002 at 06:25:10PM -0000, Tom Nixon wrote:
I am in the process of evaluating Zope and so far it looks great.
I have one question about Zope's database: How does it compare in performance to a relational database server? I am think about a scenario such as a recruitment site with a large database of jobs and candidates. Would the Zope database manage it OK or is it normal to run a database server such as mySQL alongside it for the heavy-duty stuff? Is it possible to build complex queries on the Zope database? Has anyone had any experience of building a big database-driven site in Zope, and if so did you build it all using the Zope database?
As the previous reply noted -- it is both possible and normal to run a SQL database along with ZODB. This is done by using a "Database Adaptor". There are tons of them. These include adaptors to postgres (psycopg and PoPy), Oracle, ODBC (windows only), ODBC and others (SQL-Relay), SAP, Interbase, MySQL, Sybase, Jet, Solid, and probably several I have missed. Choosing a DB is a somewhat religious issue. I belong to the PostGres religion. (and PoPy) I have tables with about 50,000 rows. I do transitive closure queries on some of them. You do want to concentrate on speed of query. You want to have enough RAM. But development is not hard at all. I find working with ZSQL methods incredibly easy, especially with DTML. Some hints: Do not be afraid of defining many special purpose queries. They are easier to test and guarantee correctness than more complicated queries. Make your queries obviously correct! Using DTML is easier for ZSQL tables, especially in the beginning, than using python methods; mostly because it does a lot of magic marshalling and unmarshalling of arguments for you. I even use SQL queries to create pull-downs and such. By using an external database and maintenance forms; I can step back from much customization. That is, many of my forms customize themselves according to the contents of a table. (This could be done with ZODB, as well) ZODB works by adding changed/new items to the end of the database and marking invalid/superceded the old item. This means that ZODB can require regular packing, or that it can grow very large very quickly. Never store things in ZODB that change often (things like page counters are prime examples.) Jim Penny
Hi Zopers, Jim Penny wrote:
Never store things in ZODB that change often (things like page counters are prime examples.)
Are there other solutions for counter-like apps than an RDBMS or the file system? (We don't wanna give Zope file system access for maximum security... ) Do I really have to set up an RDBMS and a table for small, tabular, frequently changing data? Can I do this with non_undo dbs? Is the write process slow because of the undo feature? Thanx, nuon
On Tue, Jan 29, 2002 at 06:18:00PM -0800, schimcsig marton wrote:
Hi Zopers,
Jim Penny wrote:
Never store things in ZODB that change often (things like page counters are prime examples.)
Are there other solutions for counter-like apps than an RDBMS or the file system? (We don't wanna give Zope file system access for maximum security... ) Do I really have to set up an RDBMS and a table for small, tabular, frequently changing data?
Can I do this with non_undo dbs? Is the write process slow because of the undo feature?
This is my take -- others can differ. The problem is the nature of ZODB. It works by invalidating the object in the current place, and building a new object at the end of the database. Always; even if the object is of fixed, unchanging length. Partially this gets UNDO. Partially it is a design decision that pretty much every object database implementer has made. (If objects are large, of unpredicatable size, it makes more sense to emphasize keeping the entire object in one place than it does to worry about reusing storage. This speeds retrieval hugely.) Now, in-memory is certainly possible, but the page counter will be reset to zero every time zope is reset! This means external database or filesystem is much more practical. Remarks: 1) If you are running non-Windows, this is no big deal. make sure that zope is running non-root, and that the portion of the filesystem being used for this is writable to zope's user and no one else. Not perfect, but gives you as much isolation as you are going to get anyway. (If a cracker can remotely break zope and get into your filesystem, you have lots of other worries, anyway.) [And parts of the zope hierarchy are writeable to zope's owner. In particular, the zope/var directory must be. If you are worried about security, make sure that none of the files or libs in zope/lib are writable by zope.] 2) Even if you are running windows, no big deal. For a simple page counter, or small table, you could write an External method that opens a single file that contains the data, parses it, and then rewrites it. If the file is hardcoded into the method, and the attacker cannot see the file name, and it only reads/writes a single file per small, tabular, frequently changing data, your security exposure is very small indeed. 3) Again, you have given no indication of OS. But it should be trivial to set up a RDBMS on most modern OS's. It is even easy on Windows. In my experience, you are going to want it anyway; why not think about the opportunities it gives you from the start! Just make sure that you select a RDBMS for which there is an activly developed DA. Hint: if possible, look at PostGreSQL. PoPy and pychopg are both pretty active and reasonably easy to work with (but not to install)! Jim Penny
Thanx,
nuon
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
schimcsig marton writes:
... Are there other solutions for counter-like apps than an RDBMS or the file system? (We don't wanna give Zope file system access for maximum security... ) Do I really have to set up an RDBMS and a table for small, tabular, frequently changing data? Not, if the counters need not be persistent. If they should, the state must go somewhere...
Can I do this with non_undo dbs? Yes. Is the write process slow because of the undo feature? When you have a counter as attribute of another object, than you write the whole object not just the counter.
You can make you counter a persistent object of its own, though... Dieter
From: "Tom Nixon" <tom.nixon@aim23.com>
I have one question about Zope's database: How does it compare in performance to a relational database server? I am think about a scenario such as a recruitment site with a large database of jobs and candidates.
What is "large"? The databases I have worked with both when it comes to zope and SQL have either have tables with maybe a couple of thousands items, which is quite doable with zope, or they have had several tens of millions of records and complete databases close to terabytes, which not even Sybase or MS SQL will handle very well.
Is it possible to build complex queries on the Zope database?
No, thats pretty hard to do, so it's common to put data that you want to do complex queries on in an SQL database.
Has anyone had any experience of building a big database-driven site in Zope, and if so did you build it all using the Zope database?
Ehm, well ALL zope sites are databasedriven in one form or another. Some of them are pretty big, such as zope.org.
participants (6)
-
Dieter Maurer -
Jim Penny -
Lennart Regebro -
schimcsig marton -
Thomas B. Passin -
Tom Nixon