[Zope] Thousands of Objects - how do they manifest? (newbie)

[Casey Duncan]

On Monday 06 January 2003 10:04 am, Crosbie Fitch wrote:
> I'm thinking of developing an auction system using Zope and I'm going t=
o be
> dealing with a lot of objects (if these things are represented that way
> rather than as records in a database).
>=20
> But, I'm wondering about a few things:
>=20
> 1) Are all objects in Zope persistent? ZClass instances? Python objects=
?

No, you can create regular python objects (and even ZClass instances) tha=
t=20
exist only in memory. Just don't mix in the persistent base class, or don=
't=20
mount it into another persistent object, like a folder.
=20
> 2) Are all object instances visible as entries in a Zope UI folder?

No, you can create hidden objects all over the place if you want.

> Is it
> possible to have persistent objects without them being visible? Or does=
 it
> not matter that the UI will attempt to list them all (millions or more)=
?

Regular folders are not designed for this, but there is a product call=20
BTreeFolder2 which can contain huge numbers of objects efficiently.

You can also create your own data structures using BTrees to contain=20
arbitrarily large collections of objects efficiently.
=20
> 3) If I am expecting a very large number of objects would I always be b=
etter
> off using an external database?

Possibly. It really depends on the nature of the data, the amount of writ=
e=20
concurrency, and your query requirements among other things. If the data =
maps=20
well to records and you expect many users to write to the database at the=
=20
same time and you need very flexible query/reporting then an RDBMS may be=
=20
better.

If OTOH, your objects are blobish or highly variable, its a read-heavy=20
application (most web apps are) and you have pretty straighforward query=20
requirements, then ZODB storage may be better.

That said, the current FileStorage (the default used with Zope) has a mem=
ory=20
resident index which grows linearly with the number of objects in your=20
database. This will consume a lot of memory if you have millions of objec=
ts=20
(some thing like 12-20 bytes per object). I am actually currently working=
 on=20
a solution to this though... You could also use a different type of stora=
ge=20
such as DirectoryStorage or BerkeleyStorage which do not have memory resi=
dent=20
object indexes.

 or are there circumstances where ZODB is
> quite able to look after them?
>=20
> 4) If I need to be assured of being able to backup/restore the database
> and/or rebuild the site in the event of catastrophe does an external
> database provide any greater facility/reliability for achieving this? O=
r is
> ZEO the thing to use?

ZEO is not replication/redundancy of data. It is for scalability/availabl=
ity=20
by allowing multiple app servers share the same storage.

There is a commercial product called ZRS (Zope Replication Service) which=
 can=20
be used to replicate ZODB data in real time. This can be used as an autom=
atic=20
backup as well AFAIK.

Filestorage stores everything in a single file, which can be challenging =
to=20
backup when it gets really large. Again, other storages (like=20
DirectoryStorage and BerkeleyStorage) can make this easier.

As for restoration, FileStorage is pretty resilient to data loss due to t=
he=20
fact that it only appends transactions to the file. So you generally only=
=20
loose the last few transactions if the server fails catastrophically. The=
re=20
are several python utilities that come with Zope/ZODB that can be used to=
=20
check the file consistency, recover transaction data, etc.

hth,

-Casey