[ZODB-Dev] Advice needed

[Roché Compaan]

On Sat, 2006-06-24 at 09:24 +0200, Andreas Jung wrote:
> 
> --On 24. Juni 2006 08:53:43 +0200 Roché Compaan 
> <roche at upfrontsystems.co.za> wrote:
> 
> > I am curious what other strategies besides QueueCatalog you employ? Do
> > you ever use multiple backends for your apps? How do you decide that
> > this data belongs in a relational backend? How structured must the data
> > be, or how many records must be written how often?
> >
> > I find most data is highly structured (fixed schema), but this doesn't
> > make me choose an RDMBS - the frequency of writes, concurrency and
> > record volume does.
> 
> It is often the case where you have more complex data models that really 
> require a RDBMS. It is often the case that you have to perform complex 
> queries on your data. The ZCatalog is often just too weak and too slow for 
> such apps. Our CMS (to which Dieter often refers to) uses the ZODB to store 
> large amounts of SGML data. Since the documents fits perfectly inside a 
> hierarchy the ZODB is the perfect choice. However we have some apps around 
> the CMS that provide additional functionality or use the CMS for a 
> different purpose. Some of these apps use the content from the ZODB but 
> store their metadata in postgres. One not unimportant advantage is that 
> other ppl can run their own reports etc. using a postgres client without 
> approaching me to write a script or something in Python to get requested 
> data out of Zope.
> One particular app that I have been working on uses very complex queries 
> with lots of join etc....it would be hard model to implement such queries 
> on top of the ZODB/ZCatalog. Another point is performance: this app often 
> has to perform a lot of insert/update/delete operations within one 
> transaction (up to 1000 modifications). Postgres takes perhaps 5 seconds 
> for such a complex operation. You will never reach that performance with 
> Zope...touching 1000 objects and reindexing them will take much longer 
> (unless you adjust your data modell for performance reasons to the ZODB
> needs).

Thanks for sharing this example, it confirms the usefulness of a hybrid
backend. In a way it's a sanity check for me.

> So what I want to say is: the bullheaded idea to stick everything into the 
> ZODB or into a RDBMS is just stupid. Smart people think about the data 
> storage before starting a project.

I agree wholeheartedly that it is stupid to put everything in one
backend if the data model requires a hybrid backend.

>  That also reminds me of some postings on 
> the Plone mailinglist where non-technical people put all their stuff inside 
> without knowing what's happening under the hood and wonder that everything 
> explodes or becomes horribly slow at some point.

I think the difference here is that technical people want a fundamental
understanding of what is going on under the hood so that they can
explain better what they already observe.

-- 
Roché Compaan
Upfront Systems                   http://www.upfrontsystems.co.za