Hi Michael, Michael Bernstein wrote:
It seems clear that indexing and searching are more of a botleneck than storage/retreival. Nevertheless, so far I have not heard of anyone trying to store more than 60,000 objects in a rack. I need to know if there is any reason to suspect that storage (in the ZODB) or retreival performance would suffer if the number of objects was in the hundreds of thousands or even millions.
I can't answer your question; however, I may be able to help clarify the question. The ZODB is really just a transaction manager, and an interface and contract of behaviour, for an object database. You can plug a variety of Storages into the ZODB. The default storage the Zope comes with is FileStorage -- Data.fs. There are also BerkeleyStorage, OracleStorage, DBMStorage, and others, in varying states of finishedness. So, storing things in a Rack happens in a number of stages: Your application interacts with the Rack The Rack (perhaps) stores the object persistently in its BTree The BTree is a collection of persistent ZODB objects The ZODB objects are stored as Python Pickles in a FileStorage We can consider what the effect of storing 60 000 objects is at each of these interfaces. The Rack shouldn't have a problem with 60 000 objects. I doubt a BTree would have a problem. The ZODB might not like accessing many large objects during a single transaction, as all those objects need to be in memory at once. A FileStorage should have no problem reading 60 000 stored objects. However, if these objects are changing much, your Data.fs will grow fast. In any case, you may find undo and history screens take a long time to appear. However, if you are using a Rack, you have a lot of choice about where you put your data. You can put frequently changed aspects of your data on the filesystem, and the rest in FileStorage for example. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net