[Zope] Thousands of Objects - how do they manifest? (newbie)

[sean.upton@uniontrib.com]

I have put 100s of thousands of items in to a folder before, and I would
offer the following advice, for whatever it is worth:

- Use BTreeFolder2.  If you are okay with sized batches of objects using the
default sort, this is going to help, if you code your application to use
this functionality.  If you need sorted batches of objects, use the catalog
to get these listings instead.

- ObjectManager methods like _setObject and _getObject using Zope security
machinery may slow things down a bit in a folder with lots of items (at
least it did in the old BTreeFolder, this may no longer be the case in
BTreeFolder2; the workaround for the old BTreeFolder product, which had to
load the entire list of objects as a dictionary, was to use _setOb() and
_getOb(), bypassing the security machinery).

- Backup and scalability concerns might be helped if you choose to use
DirectoryStorage and the right filesystem: http://dirstorage.sf.net.

Sean

-----Original Message-----
From: Paul Winkler [mailto:pw_lists@slinkp.com]
Sent: Monday, January 06, 2003 9:57 AM
To: zope@zope.org
Subject: Re: [Zope] Thousands of Objects - how do they manifest?
(newbie)


On Mon, Jan 06, 2003 at 03:04:39PM -0000, Crosbie Fitch wrote:
> 1) Are all objects in Zope persistent? ZClass instances? Python objects?

No, usually, and not necessarily.

Objects are persistent if their classes inherit (directly or
indirectly) from Persistent, and if the object is added to another
object which is an ObjectManager, e.g. a Folder instance.
Otherwise they are not persistent.

In order to meet these requirements, the object should be constructed
either as a ZClass or as a python Product. YOu can define and instantiate
classes in an External Method, but making them persistent would be Wrong.

The reason I said ZClass instances are "usually" persistent is that you
might add one to e.g. a TemporaryFolder whose sub-objects are
never written to disk.  But in most cases, you add instances to a Folder
and they are persistent.
 
> 2) Are all object instances visible as entries in a Zope UI folder? 

Yes.

> Is it
> possible to have persistent objects without them being visible? 

not with a standard Folder.

> Or does it
> not matter that the UI will attempt to list them all (millions or more)?

It would be very hard to use. :)
Have a look at Shane's BTreeFolder2, which is designed to
solve this kind of problem. 
http://hathaway.freezope.org/Software/BTreeFolder2

 
> 3) If I am expecting a very large number of objects would I always be
better
> off using an external database? or are there circumstances where ZODB is
> quite able to look after them?

don't know, sorry. i've never pushed a zope folder beyond 100 or so items.
My biggest ZODB at the moment contains 150727 items for a total of
2.4 GB, according to the control panel.
 
> 4) If I need to be assured of being able to backup/restore the database
> and/or rebuild the site in the event of catastrophe does an external
> database provide any greater facility/reliability for achieving this?

depends on the database I guess.
You can back up zope just by copying the ZODB, but that gets interesting
when it's very large. I've heard on this list that copying a live ZODB might
be problematic if zope writes to it while you're backing it up. I don't
know if it leads to simply missing some updates in the copy, or 
worse corruption. So I'm stuck with some downtime while making the
backup copy, which is not insignificant with 2.4 GB of data.
One strategy I'm looking at to mitigate this would be:
1) restart zope in read-only mode.
2) make a copy of the zodb.
3) restart zope in normal read-write mode.

So I'd have only two very brief outages instead of one long one.

> Or is
> ZEO the thing to use?

ZEO does not address backup issues at all.
It just allows you to run multiple Zope servers from one ZODB.

-- 

Paul Winkler
http://www.slinkp.com
Look! Up in the sky! It's PERVERT  EPSILON!
(courtesy of isometric.spaceninja.com)

_______________________________________________
Zope maillist  -  Zope@zope.org
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )