[Zope] Big folders

Albert Langer Albert.Langer@directory-designs.org
18 Jun 2000 09:59:26 +0200


This is a multi-part message in MIME format.

------=_NextPart_000_0006_01BFD8FF.ED7B3CC0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit

[AL] As well as an ETA for Btrees, in the absence of ZPatterns
documentation, could somebody please post URLs for specific references to
further information about Btrees in particular and Zpatterns in general,  -
eg email list archives and other Wickis particular thread URLs etc.

I've been studying the ZPatterns Wicki and code with a view to following up
on Kevin's earlier email reference to it which I posted to Zcommerce
recently. Didn't notice any explicit reference to Btree folders within
ZPatterns. As far as I can make out, Kevin is saying, and appears to be
right, that by designing for "Zpatterns" it should be feasible use either
ZODB or an RDBM without having to rewrite the whole application, and also
feasible to rewrite just the parts that are affected by different facilities
on different RDBMs to support more than one RDBM option. (Of course if a
Zope application is NOT designed from the start to do that, but simply
assumes the ZODB will be the only data source, this would not apply.
Unfortunately the two currently available Zope ecommerce products appear to
have taken that route).

However I got a strong impression ZPatterns may not be as useful as I
thought, in a recent exchange with Chris McDonough in an OpenACS bboard
(attached, with link to discussion). Any comments and references from Kevin
and other people more familiar with ZPatterns would be appreciated.

BTW While Btrees might make ZODB usable with very large folders I can't see
how it would make it better than using an RDBM in that situation (hopefully
under Zope, using ZPatterns). Once you have a lot more data than fits in RAM
you are likely to also have a higher ratio of writes to reads than ZODB is
"aggressively optimized" for, and need an application design that explicitly
optimizes for relational operations on subsets of data rather than
navigating from one linked object to another as OODBMs are designed to do.
Using an OODBM like ZODB would mean a full disk seek for every item, which
RDBMs are designed to avoid. Likewise if you need concurrent external access
to the same data via ODBC I don't see how ZODB Btrees would help (eg for
ecommerce links to General Ledger and fulfilment etc).

What worries me is that in the absence of actual working examples of how to
handle RDBMs with ZPatterns, I have had to recommend that someone needing an
RDBM based application (Ecommerce) use the OpenACS ecommerce module and just
kludge a way to also use Zope for other things in parallel.

I'll be discussing the specific implications for ecommerce in Zcommerce, but
perhaps here is better for info on ZPatterns and RDBMs generally?

Also, although still not sufficiently well informed, I have a strong feeling
that the features of Postgresql as an ORDBM as opposed to an ODBM or an RDBM
are particularly relevant to Zope.

As well as a normal external Python API, Postgresql has the capability to
run scripting code as well as C functions inside the database engine as a
stored procedure language and also for dynamically loaded user defined data
types and comparison operators etc. It is a true "Object" Relational DBMS
rather than a conventional RDBMs. Already implemented for Tcl. If some
Zope/Python code was put inside it, designed to work with attribute
providers and especially indexing agents and rule agents for ZPatterns, it
strikes me that the combination could be incredibly powerful.

Has there been any discussion of that, or any thoughts now?

-----Original Message-----
From: jimmie@mail.texoma.net [mailto:jimmie@mail.texoma.net]On Behalf Of
Jimmie Houchin
Sent: Sunday, June 18, 2000 7:12 AM
To: Kevin Dangoor
Cc: Zope Mailing List
Subject: Re: [Zope] Big folders

[JH]
Thanks, this is great info.

I was planning on developing with ZPatterns but haven't downloaded or
began yet. I wasn't aware that a Rack could do that. Cool.

I don't mind going to a RDBMS if necessary, but if possible I like
sticking with Zope/ZODB.

Does anyone have an eta on BTrees for ZODB?

Jimmie Houchin


Kevin Dangoor wrote:
>
> [snip]
[JH]
> > My databases will have millions of objects in what would/will be single
> > table/folders. From what I understand ZODB doesn't currently do big
> > folders well.
> >
> > So I guess my question is, should I continue with the SQL and the RDBMS
> > or is there a solution that I don't know about or understand with ZODB?
>
[KD]
> As Shane mentioned, there's a BTreeFolder in the works. Additionally, you
> can use the "Rack" from ZPatterns. If you design for a Rack, you can
switch
> to an RDBMS later on without having to change your application code. But,
> you might not need to switch to an RDBMS, because the current Rack
> implementation can store objects using BTrees within the ZODB. So,
> theoretically, you can have great big piles of objects within a single
Rack.
>
> Unfortunately, ZPatterns is still very new and there is not a lot of
> documentation. But, when you start playing with it, things become a bit
> clearer.
>
> Kevin
>

------=_NextPart_000_0006_01BFD8FF.ED7B3CC0
Content-Type: message/rfc822
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment

From: <chrism@digicool.com>
To: <Albert.Langer@Directory-Designs.Org>
Subject: Response to Article on ACS x Zope
Date: Thu, 15 Jun 2000 19:01:07 +1000
Message-ID: <200006150908.FAA23688@biont.pair.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
X-Envelope-To: <Albert.Langer@Directory-Designs.Org>
Importance: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200
X-UIDL: 4ecc648f5570f48a14103597fdc47287

Chris McDonough (chrism@digicool.com) responded to a message you
requested notification for in the OpenACS compared to other toolkits and
technologies bboard:

Subject:  Response to Article on ACS x Zope

To answer some of Albert's q's...

The ZPatterns stuff is the brainchild of Phillip Eby, a developer at Verio
(the
hosting provider).  It (among other things) is a mechanism to provide
medium-independent "propertysheet" support.  "Propertysheets" in Zope are
rectangular sets of typed data that can be attached to Zope objects
arbitrarily, and as such lend themselves very well to storage in an RDB
table.
And I think this was probably what Philip had in mind in providing Dynamic
Data
Objects.  Although to be honest, I might be totally wrong... I can't really
tell what the intent is now.  I've heard it's a good set of ideas, but
personally I'm stumped most of the time when I read his stuff because he
names
things very opaquely and seems to explain things in such a way that you *do*
need to community college C++ degree to begin to grasp it.  I think it's one
of
those projects that needs to gain some traction and obtain a critical mass
of
real-world examples in order to make its ultimate intent clear to mortals
like
me.  Until then, I can't speak much for it, although I do recommend you
continue looking at it.  (Maybe you can write some code using it that would
explain it to me!)

ZPatterns is not part of the Zope distribution.

I think Ben had in mind at least in his most recent posting an abstraction
layer that allowed RDB-savvy people to write SQL-like queries that would
automagically be translated into their equivalent optimized Oracle- or
Postgres-specific SQL dialect at runtime.  I don't think anybody in Zopeland
has the intent of writing such a beast.

You're probably right in assuming that the ZPatterns stuff is about as close
as
it gets to wanting to do something like that.  But Zope propertysheets are
generally by nature pretty small fixed sets of data (maybe 5 - 12 properties
per object).  You would need to declare some sort of table structure for a
particular object type, create the tables to store the propertysheet data
for
those kinds of objects, then do the right voodoo to tell the objects to
obtain
their properties out of SQL instead of out of the ZODB.  This could probably
be
abstracted in such a way that people don't need to write any SQL, but it
would
never approach the sort of complexity that I think Ben is talking about
because
it would probably practically necessitate an approach something along the
lines
of defining a table to hold propertysheet data for each different object
type.
If you wanted to do ad hoc queries against an existing set of data in
multiple
tables (ones that might require unions, outer joins, or insert selects)
would
need to be (just as painfully) manually constructed in the SQL dialect of
your
choice.  Personally, I'd probably try to keep it simple and create views on
existing tables that would let me operate as if I were doing so on a single
table.

Somehow I get the feeling that in this conversation we're all talking an
entirely different language than one another.  I assume that Ben's concerned
about complicated SQL abstraction between Oracle and Postgres because he
wants
to get rid of Oracle dependencies in ACS.  I think Albert's talking about
being
able to abstract the SQL out enough to not have to think about it at all
when
constructing an application.  I'm probably just plain not thinking about it
at
all.

I looked at Ben's sql-abstraction page... I don't know any tcl, but I think
I
get the gist.

It would seem to me to make sense to abstract out both 1) the access
mechanism
for querying and the 2) resultset that comes back from the query.  In Zope
we
do the first part by creating a "method" that contains the actual query
(which
can be as DB-specific or generic as you like), connecting it to a database
connection (a pool of database handles for a specific DB type).  You can
then
pass in arguments to the method that have counterpart template definitions
within the query itself (ala CUSTID = <dtml-sqlvar abc>), and the method
passes
the query in to the DB and obtains its results.

The second part (returning the results of the query to the caller) is
handled
in Zope by wrapping the resultset in a class instance that lets us process
those results generically.  It would probably be good to do something like
this
in ACS by defining a datatype that could hold arbitrary common elements that
could be processed in tcl code generically.

Though it's not a trivial engineering task, it seems like the biggest
problem
isn't implementing it, but encouraging people to use it.  From what I
understand there's a large body of existing code that does not use any sort
of
abstraction like this.  Convincing folks of the usefulness of writing
against
your interface instead of yanking the stuff right out of the tables in their
code is probably the hardest task.

-----------------

To post a response, come back to the bulletin board at

http://openacs.org/bboard/q-and-a.tcl?topic_id=8&topic=OpenACS%20compared%20
to%20other%20toolkits%20and%20technologies



-------------

If you are no longer interested in this thread, simply go to the
following URL and you will no longer get these notifications:

http://openacs.org/bboard/shut-up.tcl?row_id=1488768

-------------

Note:  this message was sent by a robot.




------=_NextPart_000_0006_01BFD8FF.ED7B3CC0
Content-Type: message/rfc822
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment

From: <Albert.Langer@Directory-Designs.Org>
To: <Albert.Langer@Directory-Designs.Org>
Subject: Response to Article on ACS x Zope
Date: Thu, 15 Jun 2000 16:13:13 +1000
Message-ID: <200006150620.CAA14750@biont.pair.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook CWS, Build 9.0.2416 (9.0.2910.0)
X-Envelope-To: <Albert.Langer@Directory-Designs.Org>
Importance: Normal
X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200
X-UIDL: 4a8af1e9599edf5996bdd7f8b8c4a6b7

Albert Langer (Albert.Langer@Directory-Designs.Org) responded to a message
you
requested notification for in the OpenACS compared to other toolkits and
technologies bboard:

Subject:  Response to Article on ACS x Zope

[CM]
There is no SQL abstraction layer in Zope like Ben's talking about.  You
still
need to write DB-specific SQL to gain results out of a query.
To my knowledge, the only system that abstracts this layer out well is
Apple's
WebObjects which I believe can use Oracle, Sybase and Informix as its object
persistence layer without requring that you  write any SQL.  We don't do
this.

We instead use the ZODB as our object persistence layer, and we let people
connect to RDB's and use that RDB's specific set of SQL commands.  To get
anything useful out of the RDB's, you need to write system-specific SQL.

[AL]
My understanding is that Ben is talking about something quite different from
WebObjects.

To clarify that, I have posted some notes about Web Objects in a separate
topic, "Apple WebObjects", on this bboard:

http://openacs.org/bboard/q-and-a-fetch-msg.tcl?msg_id=0000Hi&topic_id=8&top
ic=OpenACS%20compared%20to%20other%20toolkits%20and%20technologies

Can we all agree that something like WebObjects is beyond the scope of what
OpenACS is after for data abstraction and beyond the scope of what Zope
intends to provide?

>From my perspective the use of ZODB as an object persistent layer is fine
for many things but completely unacceptable for Ecommerce OLTP transactions.
For those you need not only to get things out of RDBMs, but also put them
into RDBMs and the application is really an RDBM application.

I don't buy the idea that Ecommerce based on ZODB alone without an RDBM is
viable, for reasons explained in 1) and 2) of:

http://lists.codeit.com/pipermail/zcommerce/2000-June/000257.html

BTW If you are aware of anybody else attempting Ecommerce OLTP with an
OODBMs please provide a reference. Otherwise please confirm that Zcommerce
efforts to do so using ZODB without an RDBM should be considered
"experimental".

Despite that, I still believe Zope could be useful together with an RDBM
based Ecommerce module and I am interested in whether it is possible for the
existing ACS Ecommerce module to be made more "stand alone" and used with
both OpenACS and Zope, or whether development would inevitably need to
"fork".

I claimed that:

For access to RDBMs databases via SQL, Zope provides pretty much exactly the
sort of data abstraction facilities that Ben was talking about in:
http://openacs.org/doc/sql-abstraction.html

Based on that view I am proposing that OpenACS people and Zope people should
get together in work on data abstraction.

Your comments below, and those from Jimmie Houchin, seem to indicate that
I'm wrong about that.

Given that you know far more about Zope than I do, I'll have to accept your
view. But before abandoning the idea I'd like to get it clear whether we are
really talking about the same thing.

[CM]
Zope does abstract out the ability to process results of SQL queries.  If
you've got a Zope system with connections to different databases (for
example
if you're grabbing stuff out of an Oracle DB and a Postgres DB within the
same
Zope instance), you create database connections to the two DB's, you create
"SQL methods" for each query for each DB, and then in Python or DTML you
pass
arguments in to the "SQL method" that houses each query, and you can process
its results generically.  You needn't worry about where the data came from.
Here's an example:

SQL method "getstatus" query:

SELECT status
FROM customer
WHERE id = <dtml-sqlvar custid>

DTML method to process the results from this query:

Customer status:  <dtml-var "getstatus(custid=1)">


The DTML method passes the value of custid in to the SQL method, gets the
result back, and displays it.

This is the extent of Zope's SQL abstraction.  You can process query results
independent of the data source by creating a "database connection" to a
specific DB (Oracle, Postgres, MySQL, Solid, ODBC, and Sybase) then creating
"SQL methods" against these database connections that accept arguments and
return results.  The content of the SQL methods is necessarily database
specific.

[AL]
Above is my understanding of what "Z SQL" offers.

But I thought the "Dynamic Data Objects" were intended to provide rather
more.

Could you please comment more specifically on the following excerpt from my
previous posting and explain what I have misunderstood:

**************
The next level currently requires some vOOdOO to understand "ZPatterns". See
especially the section on "Dynamic Data Objects" in the contents map at:

http://www.zope.org/Members/pje/Wikis/ZPatterns/HomePage/map

Suffice to say that it is NOT one of the three options that are possible
without "very OO database OO" but a fourth option made possible by OO. Yes,
you
still need RDBM specific (or site specific) SQL for optimization, because
there
is NO WAY that can be avoided. But it hides this behind umpteen layers of
framework code so that it is completely insulated from the application
domain
queries and updates and UI templates written by non-SQL developers. This
makes
life much harder for people implementing the framework (ie core Zope
developers), just as life is hard for people writing the internals of
Postgresql. But it makes life much easier for people customizing UIs, people
customizing queries and updates based on application domain models and for
people optimizing RDBMs engine access, because they can all keep out of each
other's way. (If it's the same bunny doing them all, at least they can think
about just one thing at a time).

***************

What I had in mind is that Ben's document refers to a simple template for
inserting parameters, into SQL much like what "Z SQL" provides, but with
tags like {Oracle, Postgresql} which indicate whether a particular SQL
statement is suitable for use with a particular RDBM. This would simplify
maintenance of multiple ports by reusing whatever can be reused and
providing sufficient isolation from the rest of the application for SQL
development and Tcl development to be less tightly coupled.

It seemed to me that although the "ZPatterns" framework is currently used
only for multiple login authentication (eg from LDAP directory or RDBM
instead of Zope folders in the ZODB), it is intended to be far more widely
applicable and could be used to provide data abstraction of the sort Ben is
after, by people who understand the problems of porting between different
RDBMs as well as for providing application Domain Models, Roles and facades
for people using the ZODB.



-----------------

To post a response, come back to the bulletin board at

http://openacs.org/bboard/q-and-a.tcl?topic_id=8&topic=OpenACS%20compared%20
to%20other%20toolkits%20and%20technologies



-------------

If you are no longer interested in this thread, simply go to the
following URL and you will no longer get these notifications:

http://openacs.org/bboard/shut-up.tcl?row_id=1488768

-------------

Note:  this message was sent by a robot.




------=_NextPart_000_0006_01BFD8FF.ED7B3CC0--


_______________________________________________
Zope maillist  -  Zope@zope.org
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )


---