Thanks, this is great info. I was planning on developing with ZPatterns but haven't downloaded or began yet. I wasn't aware that a Rack could do that. Cool. I don't mind going to a RDBMS if necessary, but if possible I like sticking with Zope/ZODB. Does anyone have an eta on BTrees for ZODB? Jimmie Houchin Kevin Dangoor wrote:

[snip]

...

My databases will have millions of objects in what would/will be single table/folders. From what I understand ZODB doesn't currently do big folders well.

So I guess my question is, should I continue with the SQL and the RDBMS or is there a solution that I don't know about or understand with ZODB?

As Shane mentioned, there's a BTreeFolder in the works. Additionally, you can use the "Rack" from ZPatterns. If you design for a Rack, you can switch to an RDBMS later on without having to change your application code. But, you might not need to switch to an RDBMS, because the current Rack implementation can store objects using BTrees within the ZODB. So, theoretically, you can have great big piles of objects within a single Rack.

Unfortunately, ZPatterns is still very new and there is not a lot of documentation. But, when you start playing with it, things become a bit clearer.

Kevin

_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )

----- Original Message ----- From: "Albert Langer" <Albert.Langer@Directory-Designs.org> To: "'Jimmie Houchin'" <jhouchin@texoma.net>; "'Kevin Dangoor'" <kid@kendermedia.com> Cc: "'Zope Mailing List'" <zope@zope.org> Sent: Saturday, June 17, 2000 6:33 PM Subject: RE: [Zope] Big folders

[AL] As well as an ETA for Btrees, in the absence of ZPatterns documentation, could somebody please post URLs for specific references to further information about Btrees in particular and Zpatterns in eneral, - eg email list archives and other Wickis particular thread URLs etc.

The BTrees are included with Zope itself (ZCatalog uses them). I'm not sure if there are docs online for the btrees themselves, though I did see a message on the mailing list (zope-dev, I believe) a few weeks back that gave a brief run down of what you can do with the BTree module. (The message was from Michel Pelletier).

I've been studying the ZPatterns Wicki and code with a view to following up on Kevin's earlier email reference to it which I posted to Zcommerce recently. Didn't notice any explicit reference to Btree folders within ZPatterns.

ZPatterns does not come with a "BTreeFolder"... Basically, ZPatterns includes a "Rack" object which contains "RackMountable" objects. The pattern defines a standard set of methods (like getItem, newItem) that can be implemented to pull information from different sources in a way that is transparent to other Zope components. If you look through the code in Racks.py, you'll see that the default implementation stores the objects in a BTree in the ZODB. Racks don't have the friendly user interface that a normal Folder does. But, you don't necessarily want that interface, because you wouldn't want to wait for a listing of 10,000 objects.

As far as I can make out, Kevin is saying, and appears to be right, that by designing for "Zpatterns" it should be feasible use either ZODB or an RDBM without having to rewrite the whole application, and also feasible to rewrite just the parts that are affected by different facilities on different RDBMs to support more than one RDBM option. (Of course if a Zope application is NOT designed from the start to do that, but simply assumes the ZODB will be the only data source, this would not apply. Unfortunately the two currently available Zope ecommerce products appear to have taken that route).

Correct. Using ZPatterns, you can make some parts of an object come from an RDBMS if you wish, other parts from the ZODB. And, yes, the existing ecommerce products are designed to use the ZODB alone, as far as I've seen.

However I got a strong impression ZPatterns may not be as useful as I thought, in a recent exchange with Chris McDonough in an OpenACS bboard (attached, with link to discussion). Any comments and references from Kevin and other people more familiar with ZPatterns would be appreciated.

As Chris mentioned in his message there, ZPatterns is not well documented and there is no collection of example programs to work from. So far, for many people, ZPatterns probably seems like more trouble than it's worth. But, ZPatterns 0.4 provides a bunch of new functionality that is very useful. The main intention of ZPatterns is to allow you to create very customizable Zope products. If you're developing something in-house that will be used for one product, you don't really need ZPatterns (well, you might want it because requirements *always* change over time, but you certainly don't need it to get your app going.) If you're developing some code that can be applied to many different scenarios with some small amount of customization, ZPatterns is great. LoginManager is the best living example of that. Here is a product that provides Zope authentication. Your users can login using cookies or basic auth (or even session info in the URL, if you feel like writing that code). User objects can get their data from the ZODB, an RDBMS, or LDAP. Or all three at once. There is a lot of code reuse going on there...

BTW While Btrees might make ZODB usable with very large folders I can't see how it would make it better than using an RDBM in that situation (hopefully under Zope, using ZPatterns). Once you have a lot more data than fits in RAM you are likely to also have a higher ratio of writes to reads than ZODB is "aggressively optimized" for, and need an application design that explicitly optimizes for relational operations on subsets of data rather than navigating from one linked object to another as OODBMs are designed to do. Using an OODBM like ZODB would mean a full disk seek for every item, which RDBMs are designed to avoid. Likewise if you need concurrent external access to the same data via ODBC I don't see how ZODB Btrees would help (eg for ecommerce links to General Ledger and fulfilment etc).

You are correct that the current ZODB implementation really is optimized for a high ratio of reads to writes. There is a new conflict resolution proposal that will allow you to write frequently without significant headaches. (Keeping in mind that you probably would want to use something other than FileSystem storage for something that you are writing frequently to, since FSStorage just grows and grows until it is packed.) I don't know that ZODB automatically has a "full disk seek" for every item. Zope does have an object cache, and that is actually part of how the BTrees help. If you have a Folder with 10,000 subobjects, when you access that Folder, all 10,000 subobjects get activated. With a BTreeFolder or Rack, you're only going to activate log 10000 objects to get to one of those subobjects. So, you end up with more room in your cache for useful objects, and don't read tons of info off of the disk. Mind you, I'm *not* stating that there is no place for an RDBMS. All I'm saying is that, if you like working with objects, there may be more scenarios in which the ZODB will work just fine.

What worries me is that in the absence of actual working examples of how to handle RDBMs with ZPatterns, I have had to recommend that someone needing an RDBM based application (Ecommerce) use the OpenACS ecommerce module and just kludge a way to also use Zope for other things in parallel.

Keep in mind that, depending on your needs, standard SQLMethods may do the trick. If someone is on a tight timeline for getting something done, I wouldn't recommend ZPatterns at the moment.

I'll be discussing the specific implications for ecommerce in Zcommerce, but perhaps here is better for info on ZPatterns and RDBMs generally?

My feeling is that commerce related design questions would go on the ZCommerce list... Questions about ZPatterns would go on zope-dev (since ZPatterns is still very new and not in common Zope usage).

Also, although still not sufficiently well informed, I have a strong

feeling

that the features of Postgresql as an ORDBM as opposed to an ODBM or an RDBM are particularly relevant to Zope.

I haven't read about the PostgreSQL object features at all... Kevin