I have put 100s of thousands of items in to a folder before, and I would offer the following advice, for whatever it is worth: - Use BTreeFolder2. If you are okay with sized batches of objects using the default sort, this is going to help, if you code your application to use this functionality. If you need sorted batches of objects, use the catalog to get these listings instead. - ObjectManager methods like _setObject and _getObject using Zope security machinery may slow things down a bit in a folder with lots of items (at least it did in the old BTreeFolder, this may no longer be the case in BTreeFolder2; the workaround for the old BTreeFolder product, which had to load the entire list of objects as a dictionary, was to use _setOb() and _getOb(), bypassing the security machinery). - Backup and scalability concerns might be helped if you choose to use DirectoryStorage and the right filesystem: http://dirstorage.sf.net. Sean -----Original Message----- From: Paul Winkler [mailto:pw_lists@slinkp.com] Sent: Monday, January 06, 2003 9:57 AM To: zope@zope.org Subject: Re: [Zope] Thousands of Objects - how do they manifest? (newbie) On Mon, Jan 06, 2003 at 03:04:39PM -0000, Crosbie Fitch wrote:
1) Are all objects in Zope persistent? ZClass instances? Python objects?
No, usually, and not necessarily. Objects are persistent if their classes inherit (directly or indirectly) from Persistent, and if the object is added to another object which is an ObjectManager, e.g. a Folder instance. Otherwise they are not persistent. In order to meet these requirements, the object should be constructed either as a ZClass or as a python Product. YOu can define and instantiate classes in an External Method, but making them persistent would be Wrong. The reason I said ZClass instances are "usually" persistent is that you might add one to e.g. a TemporaryFolder whose sub-objects are never written to disk. But in most cases, you add instances to a Folder and they are persistent.
2) Are all object instances visible as entries in a Zope UI folder?
Yes.
Is it possible to have persistent objects without them being visible?
not with a standard Folder.
Or does it not matter that the UI will attempt to list them all (millions or more)?
It would be very hard to use. :) Have a look at Shane's BTreeFolder2, which is designed to solve this kind of problem. http://hathaway.freezope.org/Software/BTreeFolder2
3) If I am expecting a very large number of objects would I always be better off using an external database? or are there circumstances where ZODB is quite able to look after them?
don't know, sorry. i've never pushed a zope folder beyond 100 or so items. My biggest ZODB at the moment contains 150727 items for a total of 2.4 GB, according to the control panel.
4) If I need to be assured of being able to backup/restore the database and/or rebuild the site in the event of catastrophe does an external database provide any greater facility/reliability for achieving this?
depends on the database I guess. You can back up zope just by copying the ZODB, but that gets interesting when it's very large. I've heard on this list that copying a live ZODB might be problematic if zope writes to it while you're backing it up. I don't know if it leads to simply missing some updates in the copy, or worse corruption. So I'm stuck with some downtime while making the backup copy, which is not insignificant with 2.4 GB of data. One strategy I'm looking at to mitigate this would be: 1) restart zope in read-only mode. 2) make a copy of the zodb. 3) restart zope in normal read-write mode. So I'd have only two very brief outages instead of one long one.
Or is ZEO the thing to use?
ZEO does not address backup issues at all. It just allows you to run multiple Zope servers from one ZODB. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's PERVERT EPSILON! (courtesy of isometric.spaceninja.com) _______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )