[ZODB-Dev] What are the ZopeDB limit?

[Dieter Maurer]

Michele Bendazzoli wrote at 2003-8-18 15:56 +0200:
 > I hope this is the right place to post my question. 
 > My apologies if it is not.
 > 
 > I'm going to develop a web application to manage a potentially very
 > large db using zope + postgresql.
 > ...
 > I wonder if the fact that archetypes stores the row's objects in the
 > zopedb can create future problem of performances and/or scalability
 > and/or robustness.

When you store a large number of objects in a single folder,
you must be a bit carefull:

  *  The standard "Folder" interface becomes unhandy for about 100 objects

  *  The Folder becomes inefficient for about 1000 objects.

You will need a specialized folder in this case, one derived
from "BTreeFolder2". It has been recently reported in the "plone-users"
mailing list, that Plone 1.1 will contain such a folder type.

ZCatalog, Zope's search engine, provides much less flexibility
than SQL queries. Especially, there are no joins and the stock
catalog can only and subqueries for different indexes (i.e.
you have "or" only to combine search terms for the same index).

ZCatalog, especially when you define many and/or large MetaData,
tends to bloat the ZODB. We observed that modifying a 2 kB
document lead to a transaction size of 200 kB, mainly caused
by large "MetaData" records of the ZCatalog (do not use
"MetaData", if you must, do not include "bobobase_modification_time").

Depending on your platform, you must watch out for the
2 GB limit. On most Unix platforms, you can recompile
Python to get large file support. On Windows, you can
use Python 2.2.3 (but you need a C development system to
regenerate Zope from sources for this Python version).

 > What do you think of?

When your data fits well in a relational database
(a huge number of highly structured records, no full text indexes),
put it in Postgres. Otherwise, try the ZODB.
Make some preliminary mass tests before your invest a lot in one
approach.


Dieter