On Monday 06 January 2003 10:04 am, Crosbie Fitch wrote:
I'm thinking of developing an auction system using Zope and I'm going to be dealing with a lot of objects (if these things are represented that way rather than as records in a database).
But, I'm wondering about a few things:
1) Are all objects in Zope persistent? ZClass instances? Python objects?
No, you can create regular python objects (and even ZClass instances) that exist only in memory. Just don't mix in the persistent base class, or don't mount it into another persistent object, like a folder.
2) Are all object instances visible as entries in a Zope UI folder?
No, you can create hidden objects all over the place if you want.
Is it possible to have persistent objects without them being visible? Or does it not matter that the UI will attempt to list them all (millions or more)?
Regular folders are not designed for this, but there is a product call BTreeFolder2 which can contain huge numbers of objects efficiently. You can also create your own data structures using BTrees to contain arbitrarily large collections of objects efficiently.
3) If I am expecting a very large number of objects would I always be better off using an external database?
Possibly. It really depends on the nature of the data, the amount of write concurrency, and your query requirements among other things. If the data maps well to records and you expect many users to write to the database at the same time and you need very flexible query/reporting then an RDBMS may be better. If OTOH, your objects are blobish or highly variable, its a read-heavy application (most web apps are) and you have pretty straighforward query requirements, then ZODB storage may be better. That said, the current FileStorage (the default used with Zope) has a memory resident index which grows linearly with the number of objects in your database. This will consume a lot of memory if you have millions of objects (some thing like 12-20 bytes per object). I am actually currently working on a solution to this though... You could also use a different type of storage such as DirectoryStorage or BerkeleyStorage which do not have memory resident object indexes. or are there circumstances where ZODB is
quite able to look after them?
4) If I need to be assured of being able to backup/restore the database and/or rebuild the site in the event of catastrophe does an external database provide any greater facility/reliability for achieving this? Or is ZEO the thing to use?
ZEO is not replication/redundancy of data. It is for scalability/availablity by allowing multiple app servers share the same storage. There is a commercial product called ZRS (Zope Replication Service) which can be used to replicate ZODB data in real time. This can be used as an automatic backup as well AFAIK. Filestorage stores everything in a single file, which can be challenging to backup when it gets really large. Again, other storages (like DirectoryStorage and BerkeleyStorage) can make this easier. As for restoration, FileStorage is pretty resilient to data loss due to the fact that it only appends transactions to the file. So you generally only loose the last few transactions if the server fails catastrophically. There are several python utilities that come with Zope/ZODB that can be used to check the file consistency, recover transaction data, etc. hth, -Casey