[Zope] Folder with one million Documents?

[Joachim Werner]

Hi!

> Unfortunately my documents are not static files. They are python
> classes. They contain
> fields for meta data and sometimes binary files.
> Since my documents are not files I think it is not usefull to use a
> filesystem.

But I guess the large parts are always in the binaries, and binaries can be
stored as files. Another issue is serving the objects. Will you have to
return a combination of the binary and the properties to the client? If yes,
you could still go with the caching approach that was suggested or save
pre-rendered objects to the filesystem.

 The next
> idea would be to have a database (maybe mysql, because it is fast) with
one
> table containing two colums: an ID and the pickled object.

Even if there are other opinions on this list: I can't believe that MySQL
can be THAT efficient with large files. As a matter of fact, it stores large
binaries as files. So it can't be more efficient than the file system
approach. And MySQL also will be a problem if you have a lot of concurrent
reads and writes. It is really fast with a few clients, but with many
clients PostgreSQL is supposed to skale better.

> Or you could use berkley DB for this.

The BerkeleyDB implementation of the ZODB is not really any faster AFAIK.
Just more flexible WRT building DBs that are "packless", non-undoing and the
like.

Another comment:

Why can many large docs slow down Zope? Because it will try to do things
like "objectValues", which will wake up all the children of a folderish
object. So if you can avoid these (i.e. avoid or customize the ObjectManager
API) and use BTreeFolders, many of the problems will probably go away.

I don't know about the ZCatalog (whether it is more efficient or less
efficient than doing the indexing in an RDBMS). Probably the whole thing
really has to be benchmarked properly. If we end up with the RDBMS solution
being most efficient for indexing, we might end up with a combination of all
three:

- Zope and ZODB for the "glueing" together
- RDBMS for indexing and probably storing the properties, too
- File System + Apache for serving large files

Joachim