When to use a relational database
Hi everyone, Apologies if this is a dumb question, I have had a good look through the archives and on the web. My question is basically about how to design a website. I really like the ease with which you can build applications in zope but I often see comments implying that in order to scale well one may want to use a backend relational datbase. I have also seen several comments to the effect that one should keep the state of a user session out of the ZODB. I am slightly confused by this, I can't find any information which would help me decide what data would be better off in a relational database. Example, to learn my way around zope I am deesigning a simple blogging application which allows people to log into the site and see blogs they have subscribed too. I would represent the blog posts as a series of ZPT documents. Two questions, is it better to represent the blog posts as objects in the database or as records in an external relational database? And as regards users, should I use Zope's userfolders for authentication or an external database of usernames and passwords? Again I apologize if this is a ridiculous question and if someone can point me in the direction of some good resources about the overall design of a zope site that would be very helpful. Thanks in advance -- View this message in context: http://www.nabble.com/When-to-use-a-relational-database-tp15324690p15324690.... Sent from the Zope - General mailing list archive at Nabble.com.
On 2/7/08, quizzical <quixoptic@gmail.com> wrote:
Hi everyone,
Apologies if this is a dumb question, I have had a good look through the archives and on the web. My question is basically about how to design a
It will be a good idea to look at the discusion on this topic on plone list too. Some hints here http://www.nabble.com/When-to-use-a-relational-database-to15301512s6742.html Following link was suggested in this thread http://www.upfrontsystems.co.za/Members/roche/where-im-calling-from/zodb-ben... Chetan
quizzical wrote at 2008-2-6 16:03 -0800:
... Apologies if this is a dumb question, I have had a good look through the archives and on the web. My question is basically about how to design a website. I really like the ease with which you can build applications in zope but I often see comments implying that in order to scale well one may want to use a backend relational datbase. I have also seen several comments to the effect that one should keep the state of a user session out of the ZODB.
I am slightly confused by this, I can't find any information which would help me decide what data would be better off in a relational database.
A relational database is good when you have large amounts of strictly structured data -- e.g. millions of user records where each user record has a fews fields with atomic datatypes. The ZODB is good when you have weakly structured or unstructured data and your write rate is not too high. The ZODB uses an almost self describing format. Therefore, it can store virtually everything, without any data model fixed beforehand. However, this leads to high redundancy and efficiency loss. This is not so problematic for unstructured or semi structured content -- as there is no much structure to exploit. It is also mot problematic for small amounts of data (then optimality is not an issue). But for large amounts of highly structured data, you can gain a lot with respect to space and time efficiency from a relational database. -- Dieter
I need to create a site that works similarly to php.net...the problem is the same - to document a language. From what I've seen, zope is fully capable of doing nearly everything I need it do, but I am uncertain about one aspect...check out: http://us2.php.net/manual/en/function.fopen.php for an example. The aspects of this page that I like are: 1. It is not frame based 2. The left column shows a list of functions in the same category 3. The left column allows one to move up the function tree (i.e. ^Filesystem function, ^Function Reference, etc.) 4. There is static documentation for every function (Description, Parameters, Return Values, etc. - I'm sure zope could handle this part) 5. Users are allowed to comment on the functions Assuming that Zope can do this, how difficult would it be? Can I get some pointers into the Zope documentation for how to accomplish this? Is it worth starting with something like Plone instead of with 'raw' zope? Thank you.
--On 7. Februar 2008 14:19:59 -0500 Eric Gorr <mailist@ericgorr.net> wrote:
I need to create a site that works similarly to php.net...the problem is the same - to document a language.
From what I've seen, zope is fully capable of doing nearly everything I need it do, but I am uncertain about one aspect...check out:
http://us2.php.net/manual/en/function.fopen.php
for an example.
The aspects of this page that I like are:
1. It is not frame based
2. The left column shows a list of functions in the same category
3. The left column allows one to move up the function tree (i.e. ^Filesystem function, ^Function Reference, etc.)
4. There is static documentation for every function (Description, Parameters, Return Values, etc. - I'm sure zope could handle this part)
5. Users are allowed to comment on the functions
Zope is an application and basically does not care about what you want to do with it.
Assuming that Zope can do this, how difficult would it be? Can I get some pointers into the Zope documentation for how to accomplish this? Is it worth starting with something like Plone instead of with 'raw' zope?
Plone might be a good start for such a project. -aj
On Feb 7, 2008, at 2:19 PM, Eric Gorr wrote:
I need to create a site that works similarly to php.net...the problem is the same - to document a language.
From what I've seen, zope is fully capable of doing nearly everything I need it do, but I am uncertain about one aspect...check out:
http://us2.php.net/manual/en/function.fopen.php
for an example.
The aspects of this page that I like are:
1. It is not frame based
2. The left column shows a list of functions in the same category
3. The left column allows one to move up the function tree (i.e. ^Filesystem function, ^Function Reference, etc.)
4. There is static documentation for every function (Description, Parameters, Return Values, etc. - I'm sure zope could handle this part)
5. Users are allowed to comment on the functions
Assuming that Zope can do this, how difficult would it be? Can I get some pointers into the Zope documentation for how to accomplish this? Is it worth starting with something like Plone instead of with 'raw' zope?
Thank you.
Yes, easily. I'd go with raw zope.
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists -http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Does a zope installer exist for Mac OS X or must I build from source?
--On 12. Februar 2008 12:37:19 -0500 Eric Gorr <mailist@ericgorr.net> wrote:
Does a zope installer exist for Mac OS X or must I build from source?
We only provider an installer for windows. You might look at the installers for Plone (having Zope under the hood). Or you play with paster/Zopeskel in order to build Zope with one command-line call - even on MacOSX. -aj
Dieter Maurer wrote:
quizzical wrote at 2008-2-6 16:03 -0800:
... Apologies if this is a dumb question, I have had a good look through the archives and on the web. My question is basically about how to design a website. I really like the ease with which you can build applications in zope but I often see comments implying that in order to scale well one may want to use a backend relational datbase. I have also seen several
comments
to the effect that one should keep the state of a user session out of the ZODB.
I am slightly confused by this, I can't find any information which would help me decide what data would be better off in a relational database.
A relational database is good when you have large amounts of strictly structured data -- e.g. millions of user records where each user record has a fews fields with atomic datatypes.
The ZODB is good when you have weakly structured or unstructured data and your write rate is not too high.
The ZODB uses an almost self describing format. Therefore, it can store virtually everything, without any data model fixed beforehand. However, this leads to high redundancy and efficiency loss. This is not so problematic for unstructured or semi structured content -- as there is no much structure to exploit. It is also mot problematic for small amounts of data (then optimality is not an issue). But for large amounts of highly structured data, you can gain a lot with respect to space and time efficiency from a relational database.
-- Dieter _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Thanks everyone, thats cleared up my confusion and I can get on with Zope-ing (can Zope be a verb) :) Cheers -- View this message in context: http://www.nabble.com/When-to-use-a-relational-database-tp15324690p15341765.... Sent from the Zope - General mailing list archive at Nabble.com.
----- Original Message ----- From: "Dieter Maurer" <dieter@handshake.de>
quizzical wrote at 2008-2-6 16:03 -0800:
... Apologies if this is a dumb question, I have had a good look through the archives and on the web. My question is basically about how to design a website. I really like the ease with which you can build applications in zope but I often see comments implying that in order to scale well one may want to use a backend relational datbase. I have also seen several comments to the effect that one should keep the state of a user session out of the ZODB.
I am slightly confused by this, I can't find any information which would help me decide what data would be better off in a relational database.
A relational database is good when you have large amounts of strictly structured data -- e.g. millions of user records where each user record has a fews fields with atomic datatypes.
For example pricelists etc. But you need to make join with ZODB - with hyperlink (URL1) or ZODB folder object property
The ZODB is good when you have weakly structured or unstructured data and your write rate is not too high.
ZODB effectivity is very high at tree structured data, as catalogues etc I use ZODB as catalogue container and RDB as pricelist at one travel agency site. If you want to know which, I will send you ULR to private mail and make some source view method for you. But it is written in pure DTML, few computing-intensive parts are in Python (tree structure, etc) and ZSQL to MSSQL. No sql-inject is possible thanks to DTML ;-) Near all other my webs are pure ZODB and uses SQL to store forms data only. Pure SQL driven web I use only for our own ERP, at ZODB are only DTML and bunch of ZSQL.
The ZODB uses an almost self describing format. Therefore, it can store virtually everything, without any data model fixed beforehand. However, this leads to high redundancy and efficiency loss.
It is possible to use some products like linux symlink to avoid data duplicates. Regards, JL.
From what I have read online, the things that slow zope down are primarily the numerous security checks and the large number of objects that need to be written to the zodb when you save a document. Is there a way we can manage these bottlenecks? Has anyone rated the performance of say DTML vs ZPT? For a developer like myself, I derive no benefits from zpt because the majority of my design elements are in javascript and css. So I will continue to use dtml unless zpt actually has better performance. Have performance/scalability studies been done for any of the popular zope products? I'd like to reduce the number of objects saved to the zodb if I knew how.
Thanks, Tim On 2/7/08, Dieter Maurer <dieter@handshake.de> wrote:
quizzical wrote at 2008-2-6 16:03 -0800:
... Apologies if this is a dumb question, I have had a good look through the archives and on the web. My question is basically about how to design a website. I really like the ease with which you can build applications in zope but I often see comments implying that in order to scale well one may want to use a backend relational datbase. I have also seen several comments to the effect that one should keep the state of a user session out of the ZODB.
I am slightly confused by this, I can't find any information which would help me decide what data would be better off in a relational database.
A relational database is good when you have large amounts of strictly structured data -- e.g. millions of user records where each user record has a fews fields with atomic datatypes.
The ZODB is good when you have weakly structured or unstructured data and your write rate is not too high.
The ZODB uses an almost self describing format. Therefore, it can store virtually everything, without any data model fixed beforehand. However, this leads to high redundancy and efficiency loss. This is not so problematic for unstructured or semi structured content -- as there is no much structure to exploit. It is also mot problematic for small amounts of data (then optimality is not an issue). But for large amounts of highly structured data, you can gain a lot with respect to space and time efficiency from a relational database.
-- Dieter _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
JAvascript nor CSS cannot be dynamically generated from ZPT. ZPT is thing for HTML and XML only. But with DTML you able to generate any binary data, same as with python script or external method. Please read archives few months back, here are many discussions about DTML vs. ZPT. In shorthand - ZPT is academic and Zope core developers language due some ideas about extralarge projects like CNN which are sublimed in normal praxis. DTML is faster to learn, faster to run, in shorthand is better than ZPT. FOr speed recignition, here are this mix of 1:4:10 from time of Zope 2.5.1: products or ext. methods are the fastest. DTML is 4-times slower and ZPT is 10 times slower. Python script are slightly slower than products. It is in general of course, real situations may vary. JL. ----- Original Message ----- From: "Tim Nash" <thedagdae@gmail.com>
From what I have read online, the things that slow zope down are primarily the numerous security checks and the large number of objects that need to be written to the zodb when you save a document. Is there a way we can manage these bottlenecks? Has anyone rated the performance of say DTML vs ZPT? For a developer like myself, I derive no benefits from zpt because the majority of my design elements are in javascript and css. So I will continue to use dtml unless zpt actually has better performance. Have performance/scalability studies been done for any of the popular zope products? I'd like to reduce the number of objects saved to the zodb if I knew how.
In this article: http://www.upfrontsystems.co.za/Members/roche/where-im-calling-from/zodb-ben... Roché Compaan says the following: "For an application like Plone an insertion rate of 250 objects per second results in terrible performance if you consider that storing a single document instance leads to close to 120 objects being inserted in a single transaction. And this explains why you can hardly insert more than 2 or 3 documents per second if you are uploading them into a Plone site." Why does storing a single document lead to close to 120 objects being inserted? Is it something related to plone or is this the case for all ZPT storing? Is it similarly true for the storing of DTML documents? And how can I reduce the number of inserts in my non-plone mostly DTML application? Thanks, Tim On 2/7/08, Jaroslav Lukesh <lukesh@seznam.cz> wrote:
JAvascript nor CSS cannot be dynamically generated from ZPT. ZPT is thing for HTML and XML only. But with DTML you able to generate any binary data, same as with python script or external method.
Please read archives few months back, here are many discussions about DTML vs. ZPT.
In shorthand - ZPT is academic and Zope core developers language due some ideas about extralarge projects like CNN which are sublimed in normal praxis.
DTML is faster to learn, faster to run, in shorthand is better than ZPT.
FOr speed recignition, here are this mix of 1:4:10 from time of Zope 2.5.1:
products or ext. methods are the fastest. DTML is 4-times slower and ZPT is 10 times slower. Python script are slightly slower than products.
It is in general of course, real situations may vary.
JL.
----- Original Message ----- From: "Tim Nash" <thedagdae@gmail.com>
From what I have read online, the things that slow zope down are primarily the numerous security checks and the large number of objects that need to be written to the zodb when you save a document. Is there a way we can manage these bottlenecks? Has anyone rated the performance of say DTML vs ZPT? For a developer like myself, I derive no benefits from zpt because the majority of my design elements are in javascript and css. So I will continue to use dtml unless zpt actually has better performance. Have performance/scalability studies been done for any of the popular zope products? I'd like to reduce the number of objects saved to the zodb if I knew how.
Thanks Dieter. I will look into using the zope 3 views in 2.10 and compare their performance with DTML. BTW, I hope plone people understand that I am a big fan of plone, zope 3 and zpt. They just don't fit my current application. @quizzical: As others have pointed out, zope really excels at handling semi-structured data. In addition, zope 2 (and maybe 3) is easier to administer than a relational database so it is a database that you can distribute to power users. Tim On Feb 8, 2008 1:32 PM, Dieter Maurer <dieter@handshake.de> wrote:
Tim Nash wrote at 2008-2-7 19:14 -0800:
... Why does storing a single document lead to close to 120 objects being inserted?
Almost surely due to full text indexing.
-- Dieter
On Thu, Feb 07, 2008 at 07:14:46PM -0800, Tim Nash wrote:
Why does storing a single document lead to close to 120 objects being inserted?
I'd bet almost all of that is updates to catalog indexes and metadata.
Is it something related to plone or is this the case for all ZPT storing? Is it similarly true for the storing of DTML documents?
You're confused. The relative speed of ZPT and DTML have only to do with rendering time. As far as the ZODB is concerned, they are equivalent. -- Paul Winkler http://www.slinkp.com
On Friday 08 February 2008, Dieter Maurer wrote:
Jaroslav Lukesh wrote at 2008-2-7 22:40 +0100:
JAvascript nor CSS cannot be dynamically generated from ZPT.
They can be generated by ZPTs -- though it is not very natural.
Have a look at z3c.zrtresource, which is a simple package allowing some variable substitution. It was designed for CSS and JS. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"
Tim Nash wrote at 2008-2-7 13:21 -0800:
From what I have read online, the things that slow zope down are primarily the numerous security checks
In modern Zope versions you can use Zope 3 views. The corresponding templates are trusted. For earlier Zope versions, there is "TrustedExecutables" (they do not yet work for Zope 2.10).
and the large number of objects that need to be written to the zodb when you save a document.
I doubt that the large number of objects is the real reason. The large number of objects result from full text indexing: the object is inserted into the document list for each word it contains. However, full text indexing is slow and expensive - independent from the insertion into the document lists. Obtain a profile and check whether this is really true....
... Has anyone rated the performance of say DTML vs ZPT?
DTML is likely to be faster: it has a C implementation and is simpler than ZPT. I expect a factor of about 4. However, very often the total time is not dominated by the rendering itself but e.g. determining the values that should be rendered. -- Dieter
participants (10)
-
Andreas Jung -
Chetan Kumar -
Dieter Maurer -
Eric Gorr -
Jaroslav Lukesh -
Paul Winkler -
quizzical -
Stephan Richter -
Tim Nash -
Tom Von Lahndorff