[Zope] Hardware for Zope + ZEO
Wankyu Choi
wankyu@neoqst.com
Thu, 27 Mar 2003 05:24:15 +0900
Hi all,
Not sure if anyone is still interested in this thread ;-)
I tried what Toby suggested, and indeed he was right:
- Installed Zope 2.6.1 with the new Catalog code. ( impressive =
performance
boost ^^ ) + Python 2.2
- Created a NeoBoard with 5,000 heavy articles containing heavy =
attachments
( simply creating this many articles took me more than a day ).
- Changed the internals of NeoBoard to use ZCatalog when creating the
threaded article list.
The major problem with this scheme was how to sort the result set in a =
such
a way that NeoBoard would still retain its threaded look. For a couple =
of
reasons, I need two sort keys: one to sort the threads (parents) and the
other to sort their replies (childs). ZCatalog supports a single sort =
index,
so I had to come up with my own solution: slicing the result set. Since =
we
always need only a part of the whole result set that needs to be viewed =
in
the current request, NeoBoard only needs to sort that portion.=20
Admittedly, the previous versions of NeoBoard couldn't handle that many
articles without hogging the resources. After changing the internals, =
it
pulls and sorts 5,000 articles ( brains to be exact ) and displays part =
of
them ( a pageful of them ) in less than a second even on my P4 desktop. =
(
still not so fast as the SQL version, though ;-(
Regarding the hardware...
I bought two servers: a 2-way Xeon server with a raid device (MegaRAID
dual-channel controller/raid level 5 ) and a 2-way tualatin P3 server.
The Xeon server runs ZEO with Toby's DirectoryStorage ( ReiserFS on =
290+G
raid 5 -- 72G SCSI HDD * 6 --) with the tualatin server being a ZEO =
client.
Will add two more ZEO clients to this setup.
Guess it is a viable setup for the needs I mentioned in this thread. So =
far,
so good. Pretty fast and pretty stable.
Thanks for the great product and tips, Toby. I also want to thank the =
others
who shared their ideas in this thread.
All the best,
Wankyu Choi
-----Original Message-----
From: sean.upton@uniontrib.com [mailto:sean.upton@uniontrib.com]=20
Sent: Saturday, February 01, 2003 2:40 AM
To: wankyu@neoqst.com; tdickenson@geminidataloggers.com;
mvalente@ruido-visual.pt; zope@zope.org
Subject: RE: [Zope] Hardware for Zope + ZEO
Toby is right about using the Catalog, I think (or some other form of
indexing). The ObjectManager methods are not going to scale well =
because:
1 - An ObjectManager/Folder has no way of batching small sets of the =
large
store.
a. BTreeFolder 2 can batch, but only in its default sort (see 2)
2 - An ObjectManager/Folder has no way of sorting, unless you do it =
after
the fact in presentation logic (slow/bloated).
You should likely be using the catalog, and create a string-based =
FieldIndex
containing the absolute physical path of the folder, and indexes for all =
the
fields you might sort on. Batching with a Catalog query should just be =
a
matter of getting a slice of the returned list of Brain objects. You =
can
write a wrapper around this that will just get the objects those Brains
refer to anyway, since it is likely your application will need them. =20
What I might suggest is writing a simple product that creates a
Catalog-backed folder: it effeciently returns sorted slices of all or a
subset of objects in the folder, encapsulating access to the Catalog, =
and
creating your own API for a slightly smarter folder. So you would be =
adding
some smart methods on top of the existing dumb methods in the =
ObjectManager
interface that the folder already exposes. Then, each folder can have =
its
own Catalog specifically to act as a sorting engine, and its own =
batching
code that uses the sorting engine. I'm not sure what the performance
implications of this are for your application on write (yet another =
Catalog
to write to), but the implications on read performance should improve, =
given
what your application does.
Note: supposedly, there are eventually going to be improvements to the
ZCatalog sorting code (in Zope 2.7) that will improve its performance in
general by some great degree.
Sean
-----Original Message-----
From: Wankyu Choi [mailto:wankyu@neoqst.com]
Sent: Friday, January 31, 2003 3:48 AM
To: 'Toby Dickenson'; 'Mario Valente'; zope@zope.org
Subject: RE: [Zope] Hardware for Zope + ZEO
> Is there a way in Zope to limit the returned
> results as a limit clause would do in an SQL statement?
>>The best approach is to use ZCatalog. ZCatalog means you only need to
>>load
the=20
>>articles into memory when you need them.
A threaded board needs to maintain its threaded structure. It's rather
cumbersome with SQL but efficient. In Zope, it's a snap but gives poor
performance. A folder (parent article) and its subfolders (its replies)
strucutre directly turns into a thread hierarchy.
I do use ZCatalog for all views of NeoBoard except for its default =
threaded
look. Maybe I was stupid in not using ZCatalog for the threaded look. =
I'll
give it a try.
But I just wanted to know if there was a way to pull a limited set of
objects from a **folderish object** not using ZCatalog.=20
Not sure if your suggestion would work with NeoBoard yet, but I ran into =
a
number of situations where I should pull all objects from a folder.
I posted a simliar query months ago when I was trying to write my own =
user
folder. No definate answer yet. The stock userfolder, for example, =
returns
every user object no matter how many there are in response to a getUsers
method call. I wouldn't use it with my 300,000 users. Nobody would, I
suspect. There was no info on how to pull a limited set of users from a =
user
folder. And guess we can't use ZCatalog to do the job. Simply put, are =
Zope
folderish objects including BTreeFolderish objects **not** designed to =
hold
this many objects? Well, if so, I'll just rewrite NeoBoard to use MySQL
again. That would be a lot easier :(
Thanks again for your time.
Regards,
Wankyu Choi
---------------------------------------------------------------
To the dedicated staff at NeoQuest, language is not a problem
to be dealt with, but an art waiting to be performed.
---------------------------------------------------------------
Wankyou Choi
CEO/President
NeoQuest Communications, Inc.
3rd Floor, HMC Bldg., 730-14, Yoksam-dong, Kangnam-gu
Seoul, Korea
Tel: 82-2 - 501 - 7124 Fax: 82-2-501-7058
Corporate Home: http://www.neoqst.com
Personal Home: http://www.zoper.net, http://www.neoboard.net
e-mail: wankyu@neoqst.com
--------------------------------------------------------------- =20
_______________________________________________
Zope maillist - Zope@zope.org =
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
_______________________________________________
Zope maillist - Zope@zope.org =
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -=20
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )