[Zope] Hardware for Zope + ZEO

Sat, 1 Feb 2003 15:02:11 +0900

Well, that seems to be the answer I've been looking for ;-)

Thanks a lot for your comments and ideas. Guess I'll be writing a =
"smarter"
folder then :-)

Best Regards,
Wankyu Choi
---------------------------------------------------------------
      To the dedicated staff at NeoQuest, language is not a problem
          to be dealt with, but an art waiting to be performed.
  ---------------------------------------------------------------
  Wankyou Choi
  CEO/President
  NeoQuest Communications, Inc.
  3rd Floor, HMC Bldg., 730-14, Yoksam-dong, Kangnam-gu
  Seoul, Korea
  Tel: 82-2 - 501 - 7124 Fax: 82-2-501-7058
  Corporate Home: http://www.neoqst.com
  Personal  Home: http://www.zoper.net, http://www.neoboard.net
  e-mail:   wankyu@neoqst.com
--------------------------------------------------------------- =20

-----Original Message-----
From: sean.upton@uniontrib.com [mailto:sean.upton@uniontrib.com]=20
Sent: Saturday, February 01, 2003 2:40 AM
To: wankyu@neoqst.com; tdickenson@geminidataloggers.com;
mvalente@ruido-visual.pt; zope@zope.org
Subject: RE: [Zope] Hardware for Zope + ZEO

Toby is right about using the Catalog, I think (or some other form of
indexing). The ObjectManager methods are not going to scale well =
because:

1 - An ObjectManager/Folder has no way of batching small sets of the =
large
store.
	a. BTreeFolder 2 can batch, but only in its default sort (see 2)

2 - An ObjectManager/Folder has no way of sorting, unless you do it =
after
the fact in presentation logic (slow/bloated).

You should likely be using the catalog, and create a string-based =
FieldIndex
containing the absolute physical path of the folder, and indexes for all =
the
fields you might sort on.  Batching with a Catalog query should just be =
a
matter of getting a slice of the returned list of Brain objects.  You =
can
write a wrapper around this that will just get the objects those Brains
refer to anyway, since it is likely your application will need them. =20

What I might suggest is writing a simple product that creates a
Catalog-backed folder: it effeciently returns sorted slices of all or a
subset of objects in the folder, encapsulating access to the Catalog, =
and
creating your own API for a slightly smarter folder.  So you would be =
adding
some smart methods on top of the existing dumb methods in the =
ObjectManager
interface that the folder already exposes.  Then, each folder can have =
its
own Catalog specifically to act as a sorting engine, and its own =
batching
code that uses the sorting engine.  I'm not sure what the performance
implications of this are for your application on write (yet another =
Catalog
to write to), but the implications on read performance should improve, =
given
what your application does.

Note: supposedly, there are eventually going to be improvements to the
ZCatalog sorting code (in Zope 2.7) that will improve its performance in
general by some great degree.

Sean

-----Original Message-----
From: Wankyu Choi [mailto:wankyu@neoqst.com]
Sent: Friday, January 31, 2003 3:48 AM
To: 'Toby Dickenson'; 'Mario Valente'; zope@zope.org
Subject: RE: [Zope] Hardware for Zope + ZEO

> Is there a way in Zope to limit the returned
> results as a limit clause would do in an SQL statement?

>>The best approach is to use ZCatalog. ZCatalog means you only need to=20
>>load
the=20
>>articles into memory when you need them.

A threaded board needs to maintain its threaded structure. It's rather
cumbersome with SQL but efficient. In Zope, it's a snap but gives poor
performance. A folder (parent article) and its subfolders (its replies)
strucutre directly turns into a thread hierarchy.

I do use ZCatalog for all views of NeoBoard except for its default =
threaded
look.  Maybe I was stupid in not using ZCatalog for the threaded look. =
I'll
give it a try.

But I just wanted to know if there was a way to pull a limited set of
objects from a **folderish object** not using ZCatalog.=20

Not sure if your suggestion would work with NeoBoard yet, but I ran into =
a
number of situations where I should pull all objects from a folder.

I posted a simliar query months ago when I was trying to write my own =
user
folder. No definate answer yet.  The stock userfolder, for example, =
returns
every user object no matter how many there are in response to a getUsers
method call.  I wouldn't use it with my 300,000 users. Nobody would, I
suspect. There was no info on how to pull a limited set of users from a =
user
folder. And guess we can't use ZCatalog to do the job. Simply put, are =
Zope
folderish objects including BTreeFolderish objects **not** designed to =
hold
this many objects?  Well, if so, I'll just rewrite NeoBoard to use MySQL
again. That would be a lot easier :(

Thanks again for your time.

Regards,
Wankyu Choi
---------------------------------------------------------------
      To the dedicated staff at NeoQuest, language is not a problem
          to be dealt with, but an art waiting to be performed.
  ---------------------------------------------------------------
  Wankyou Choi
  CEO/President
  NeoQuest Communications, Inc.
  3rd Floor, HMC Bldg., 730-14, Yoksam-dong, Kangnam-gu
  Seoul, Korea
  Tel: 82-2 - 501 - 7124 Fax: 82-2-501-7058
  Corporate Home: http://www.neoqst.com
  Personal  Home: http://www.zoper.net, http://www.neoboard.net
  e-mail:   wankyu@neoqst.com
--------------------------------------------------------------- =20

_______________________________________________
Zope maillist  -  Zope@zope.org =
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -  http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )