In an effort to put our Zope-based system into production, I ran an import script to pull in historical data external to Zope. At this point we are experiencing some very strange behaviors - seemingly related to traversal of the object tree - that did not occur prior to the import. At this point we have close to 150,000 objects, each containing a handful of metadata and nothing more. A large number of these are folderish. I expect this database to grow much larger in a very short time, providing it does not die a premature death. Our file system is Reiser FS and our ZODB is running on DirectoryStorage, rather than the Data.fs file. I worry that we're pushing Zope beyond its capabilities with this number of objects, but I don't really know. Does anyone have experience with databases of this size? Any ideas on limitations to number of objects the ZODB can comfortably manage, or the size to which the ZODB can comfortably grow?
--On Dienstag, 14. Oktober 2003 12:24 Uhr -0400 nwingfield@che-llp.com wrote:
In an effort to put our Zope-based system into production, I ran an import script to pull in historical data external to Zope. At this point we are experiencing some very strange behaviors - seemingly related to traversal of the object tree - that did not occur prior to the import. At this point we have close to 150,000 objects, each containing a handful of metadata and nothing more. A large number of these are folderish. I expect this database to grow much larger in a very short time, providing it does not die a premature death. Our file system is Reiser FS and our ZODB is running on DirectoryStorage, rather than the Data.fs file. I worry that we're pushing Zope beyond its capabilities with this number of objects, but I don't really know. Does anyone have experience with databases of this size? Any ideas on limitations to number of objects the ZODB can comfortably manage, or the size to which the ZODB can comfortably grow?
150.000? That's what I call a medium-sized ZODB. I might worry about the ZODB when the number of objects reaches 2-3 million objects. ZODB storages with a size of several 20-30 are known to work. Although packing and backup might be a bit tricky. You might think about using DirectoryStorage which is much easier to backup but several times slower on packing compared to FileStorage. -aj
nwingfield@che-llp.com wrote at 2003-10-14 12:24 -0400:
In an effort to put our Zope-based system into production, I ran an import script to pull in historical data external to Zope. At this point we are experiencing some very strange behaviors - seemingly related to traversal of the object tree - that did not occur prior to the import. At this point we have close to 150,000 objects, each containing a handful of metadata and nothing more.
Hopefully, you are using efficient data structures (tree like) for you internal nodes. The standard "OFS.Folder" (unlike "BTreeFolder2.BTreeFolder") is not suitable to store several 10-thousand objects (its UI is even unsuitable to handle several hundred objects). Dieter
participants (3)
-
Andreas Jung -
Dieter Maurer -
nwingfield＠che-llp.com