[Zope] Big folders

[Kevin Dangoor]

----- Original Message -----
From: "Albert Langer" <Albert.Langer@Directory-Designs.org>
To: "'Jimmie Houchin'" <jhouchin@texoma.net>; "'Kevin Dangoor'"
<kid@kendermedia.com>
Cc: "'Zope Mailing List'" <zope@zope.org>
Sent: Saturday, June 17, 2000 6:33 PM
Subject: RE: [Zope] Big folders


> [AL] As well as an ETA for Btrees, in the absence of ZPatterns
> documentation, could somebody please post URLs for specific references to
> further information about Btrees in particular and Zpatterns in
eneral,  -
> eg email list archives and other Wickis particular thread URLs etc.

The BTrees are included with Zope itself (ZCatalog uses them). I'm not sure
if there are docs online for the btrees themselves, though I did see a
message on the mailing list (zope-dev, I believe) a few weeks back that gave
a brief run down of what you can do with the BTree module. (The message was
from Michel Pelletier).

> I've been studying the ZPatterns Wicki and code with a view to following
up
> on Kevin's earlier email reference to it which I posted to Zcommerce
> recently. Didn't notice any explicit reference to Btree folders within
> ZPatterns.

ZPatterns does not come with a "BTreeFolder"... Basically, ZPatterns
includes a "Rack" object which contains "RackMountable" objects. The pattern
defines a standard set of methods (like getItem, newItem) that can be
implemented to pull information from different sources in a way that is
transparent to other Zope components. If you look through the code in
Racks.py, you'll see that the default implementation stores the objects in a
BTree in the ZODB.

Racks don't have the friendly user interface that a normal Folder does. But,
you don't necessarily want that interface, because you wouldn't want to wait
for a listing of 10,000 objects.

> As far as I can make out, Kevin is saying, and appears to be
> right, that by designing for "Zpatterns" it should be feasible use either
> ZODB or an RDBM without having to rewrite the whole application, and also
> feasible to rewrite just the parts that are affected by different
facilities
> on different RDBMs to support more than one RDBM option. (Of course if a
> Zope application is NOT designed from the start to do that, but simply
> assumes the ZODB will be the only data source, this would not apply.
> Unfortunately the two currently available Zope ecommerce products appear
to
> have taken that route).

Correct. Using ZPatterns, you can make some parts of an object come from an
RDBMS if you wish, other parts from the ZODB. And, yes, the existing
ecommerce products are designed to use the ZODB alone, as far as I've seen.

> However I got a strong impression ZPatterns may not be as useful as I
> thought, in a recent exchange with Chris McDonough in an OpenACS bboard
> (attached, with link to discussion). Any comments and references from
Kevin
> and other people more familiar with ZPatterns would be appreciated.

As Chris mentioned in his message there, ZPatterns is not well documented
and there is no collection of example programs to work from. So far, for
many people, ZPatterns probably seems like more trouble than it's worth.
But, ZPatterns 0.4 provides a bunch of new functionality that is very
useful.

The main intention of ZPatterns is to allow you to create very customizable
Zope products. If you're developing something in-house that will be used for
one product, you don't really need ZPatterns (well, you might want it
because requirements *always* change over time, but you certainly don't need
it to get your app going.) If you're developing some code that can be
applied to many different scenarios with some small amount of customization,
ZPatterns is great. LoginManager is the best living example of that. Here is
a product that provides Zope authentication. Your users can login using
cookies or basic auth (or even session info in the URL, if you feel like
writing that code). User objects can get their data from the ZODB, an RDBMS,
or LDAP. Or all three at once. There is a lot of code reuse going on
there...

> BTW While Btrees might make ZODB usable with very large folders I can't
see
> how it would make it better than using an RDBM in that situation
(hopefully
> under Zope, using ZPatterns). Once you have a lot more data than fits in
RAM
> you are likely to also have a higher ratio of writes to reads than ZODB is
> "aggressively optimized" for, and need an application design that
explicitly
> optimizes for relational operations on subsets of data rather than
> navigating from one linked object to another as OODBMs are designed to do.
> Using an OODBM like ZODB would mean a full disk seek for every item, which
> RDBMs are designed to avoid. Likewise if you need concurrent external
access
> to the same data via ODBC I don't see how ZODB Btrees would help (eg for
> ecommerce links to General Ledger and fulfilment etc).

You are correct that the current ZODB implementation really is optimized for
a high ratio of reads to writes. There is a new conflict resolution proposal
that will allow you to write frequently without significant headaches.
(Keeping in mind that you probably would want to use something other than
FileSystem storage for something that you are writing frequently to, since
FSStorage just grows and grows until it is packed.)

I don't know that ZODB automatically has a "full disk seek" for every item.
Zope does have an object cache, and that is actually part of how the BTrees
help. If you have a Folder with 10,000 subobjects, when you access that
Folder, all 10,000 subobjects get activated. With a BTreeFolder or Rack,
you're only going to activate log 10000 objects to get to one of those
subobjects. So, you end up with more room in your cache for useful objects,
and don't read tons of info off of the disk.

Mind you, I'm *not* stating that there is no place for an RDBMS. All I'm
saying is that, if you like working with objects, there may be more
scenarios in which the ZODB will work just fine.

> What worries me is that in the absence of actual working examples of how
to
> handle RDBMs with ZPatterns, I have had to recommend that someone needing
an
> RDBM based application (Ecommerce) use the OpenACS ecommerce module and
just
> kludge a way to also use Zope for other things in parallel.

Keep in mind that, depending on your needs, standard SQLMethods may do the
trick. If someone is on a tight timeline for getting something done, I
wouldn't recommend ZPatterns at the moment.

> I'll be discussing the specific implications for ecommerce in Zcommerce,
but
> perhaps here is better for info on ZPatterns and RDBMs generally?

My feeling is that commerce related design questions would go on the
ZCommerce list... Questions about ZPatterns would go on zope-dev (since
ZPatterns is still very new and not in common Zope usage).

>
> Also, although still not sufficiently well informed, I have a strong
feeling
> that the features of Postgresql as an ORDBM as opposed to an ODBM or an
RDBM
> are particularly relevant to Zope.

I haven't read about the PostgreSQL object features at all...

Kevin