[Zope] Hardware for Zope + ZEO

[sean.upton@uniontrib.com]

Toby is right about using the Catalog, I think (or some other form of
indexing). The ObjectManager methods are not going to scale well because:

1 - An ObjectManager/Folder has no way of batching small sets of the large
store.
	a. BTreeFolder 2 can batch, but only in its default sort (see 2)

2 - An ObjectManager/Folder has no way of sorting, unless you do it after
the fact in presentation logic (slow/bloated).

You should likely be using the catalog, and create a string-based FieldIndex
containing the absolute physical path of the folder, and indexes for all the
fields you might sort on.  Batching with a Catalog query should just be a
matter of getting a slice of the returned list of Brain objects.  You can
write a wrapper around this that will just get the objects those Brains
refer to anyway, since it is likely your application will need them.  

What I might suggest is writing a simple product that creates a
Catalog-backed folder: it effeciently returns sorted slices of all or a
subset of objects in the folder, encapsulating access to the Catalog, and
creating your own API for a slightly smarter folder.  So you would be adding
some smart methods on top of the existing dumb methods in the ObjectManager
interface that the folder already exposes.  Then, each folder can have its
own Catalog specifically to act as a sorting engine, and its own batching
code that uses the sorting engine.  I'm not sure what the performance
implications of this are for your application on write (yet another Catalog
to write to), but the implications on read performance should improve, given
what your application does.

Note: supposedly, there are eventually going to be improvements to the
ZCatalog sorting code (in Zope 2.7) that will improve its performance in
general by some great degree.

Sean

-----Original Message-----
From: Wankyu Choi [mailto:wankyu@neoqst.com]
Sent: Friday, January 31, 2003 3:48 AM
To: 'Toby Dickenson'; 'Mario Valente'; zope@zope.org
Subject: RE: [Zope] Hardware for Zope + ZEO



> Is there a way in Zope to limit the returned
> results as a limit clause would do in an SQL statement?

>>The best approach is to use ZCatalog. ZCatalog means you only need to load
the 
>>articles into memory when you need them.

A threaded board needs to maintain its threaded structure. It's rather
cumbersome with SQL but efficient. In Zope, it's a snap but gives poor
performance. A folder (parent article) and its subfolders (its replies)
strucutre directly turns into a thread hierarchy.

I do use ZCatalog for all views of NeoBoard except for its default threaded
look.  Maybe I was stupid in not using ZCatalog for the threaded look. I'll
give it a try.

But I just wanted to know if there was a way to pull a limited set of
objects from a **folderish object** not using ZCatalog. 

Not sure if your suggestion would work with NeoBoard yet, but I ran into a
number of situations where I should pull all objects from a folder.

I posted a simliar query months ago when I was trying to write my own user
folder. No definate answer yet.  The stock userfolder, for example, returns
every user object no matter how many there are in response to a getUsers
method call.  I wouldn't use it with my 300,000 users. Nobody would, I
suspect. There was no info on how to pull a limited set of users from a user
folder. And guess we can't use ZCatalog to do the job. Simply put, are Zope
folderish objects including BTreeFolderish objects **not** designed to hold
this many objects?  Well, if so, I'll just rewrite NeoBoard to use MySQL
again. That would be a lot easier :(

Thanks again for your time.

Regards,
Wankyu Choi
---------------------------------------------------------------
      To the dedicated staff at NeoQuest, language is not a problem
          to be dealt with, but an art waiting to be performed.
  ---------------------------------------------------------------
  Wankyou Choi
  CEO/President
  NeoQuest Communications, Inc.
  3rd Floor, HMC Bldg., 730-14, Yoksam-dong, Kangnam-gu
  Seoul, Korea
  Tel: 82-2 - 501 - 7124 Fax: 82-2-501-7058
  Corporate Home: http://www.neoqst.com
  Personal  Home: http://www.zoper.net, http://www.neoboard.net
  e-mail:   wankyu@neoqst.com
---------------------------------------------------------------  


_______________________________________________
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )