On Thursday 04 Jul 2002 3:57 pm, Thierry Florac wrote:
I don't want to execute queries on Zope's database, but what I'd like to be able to do is to use an external storage, which could provide advanced functions such as :
It looks like your concerns are the same as what prompted me to start on DirectoryStorage.....
- replication (also available with ZEO),
ZEO doesnt do replication today. It is planned, but for far in the future.
- partitionning,
of what, and why?
- handling of very high volumes,
As in file size, or throughput? FileStorage leads the throughput benchmarks, but even the slowest storage is not sufficiently slow to make a difference (IMO). All of today's production storages have serious problems when packing databases with many objects, because they all use an in-memory index of reachable objects. For high volume you need a non-versioning storage (or DirectoryStorage, when its ready)
In fact, I'm a little bit afraid by the ZODB FileStorage, and by it's potential ability to handle very big data files correctly... We have a lot of big files to publish and index, and our ZODB will probably have a size of 10 GB quite rapidly... Any advise ??
Ive never had a filestorage that big. If memory usage is linear (which I think is a reasonable assumption) then a FileStorage that big will need (roughly) 1G of ram for normal use, and 2-3G during packing. If thats not a problem; do it.