[Zope-dev] ZCatalog and Huge Customized Objects in ZODB

[R. David Murray]

On Tue, 23 May 2000, Thomas Weholt wrote:
> Is it possible to store lots of huge objects, like 2-3 MB in size each, in
> ZODB and catalog them for easy searching? Can I catalog special properties
> of custom objects and skip others? Are there any limitations to ZCatalog and
> the amount of stuff to index, before the speed is affected?

Yes, Yes, and not really.  ZCatalog uses  BTrees to store the index data,
so it would take a *huge* amount of data to slow it down on searches.
If your objects and their properties are static you shouldn't have
any problem.  If either are dynamic, you are going to have a file size
growth problem with the ZODB as those large objects get written to disk
on every update.  You can get around the latter by structuring your
site so that the content is in one object and the properties are in
another helper object, or by using ZPatterns if you are willing to 
work with developing code and either wait for the indexing support
or roll your own.

Storing many large objects, though, you need to be aware of any
limits your OS puts on maximum file size.  The ZODB uses 64 bit
pointers, but, eg, Linux currently has a maximum file size of 2GB.
FreeBSD, for the reverse example, supports files up to the maximum
ZODB will handle.  Or you could work out a way to store the object
content in the file system by using external methods or LocalFS.

As for the indexing of properties, you tell ZCatalog the property
names to index (could be custom methods, too), and any cataloged
object that has the property will be indexed, and any cataloged
object that doesn't have that property will be ignored in that index.

> If somebody has pointers on how to store custom python-objects in ZODB and
> make them ZCatalog-aware or somehow work with the ZCatalog, I`d be most
> grateful.

There are a couple of How-Tos on using the catalog.  If you are
implementing your objects in Python (as opposed to ZClasses), you
just need to include CatalogAware in the base clases, and make sure
you put a self.reindex_object() call in any methods that handle
property changes.  It's very simple to do:

from Products.ZCatalog.CatalogAwareness import CatalogAware

class MyClass(CatalogAware):
    ....

    def instance_Edit(self,REQUEST):
       ....
       self.reindex_object()
       ....

--RDM