[Zope] Search on ZCatalog or ZSQL in MySQL

[Dieter Maurer]

Nestor A. Diaz L. writes:
 > I have to make a system to store news, so i have to migrate a old system
 > using perl cgi's and perform a search on text files to use Zope as the
 > search engine however i don't know if i have to use a SQL database like
 > MySQL or use the ZCatalog, i need full text search, and ZCatalog fits on
 > the true decision however the current database news is about 350M and i
 > would like to extend it to store images and file, so is possible that at
 > the final year if everyting comes ok it outperforms the 1GB data
If you have a database or search engine that handles full text,
I would go for it. However, most relational databases have difficulties
with full text. I doubt, that MySQL can handle it well.

The ZCatalog as it is now, is nice for small to medium size
content. I, personally, would not use it for content bigger
than 50 to 100 MB. Due to some weeknesses in the BTree
implementation (which is used by ZCatalog to maintain its
indexes) a considerable part of the index structure
is rewritten if the index changes. This may lead to a fast
growing ZODB with the need for frequent "pack"'s.

Before you decide for ZODB, you should make some tests:
something like this:

  *  index about 10 MB of content

  *  change content

  *  search with the catalog

  *  index another 10 MB of content (you now have 20 MB content)

  *  change content

  *  search with the catalog

For each step:
  
  Look how the size of the ZODB changes: before and after packing

  Look how the main memory requirements changes (especially
  during indexing)

  Look how the timings changes (for all relevant operations:
  indexing, packing, searching)


Try to extrapolate your results.
You may need to add more 10 MB content chunks to recognize a
tendancy.


Dieter