[Zope] Hardware for Zope + ZEO

[Wankyu Choi]

On Friday 31 January 2003 7:17 am, Wankyu Choi wrote:

> While we're at the subject... I'd be pulling, say, 10,000 objects from =

> directory storage in one go if a NeoBoard forum gets heavily populated =

> ( 10,000 articles or 5,000 articles with each article having one=20
> attachment object ).

>>Either I misunderstand this statement, or your application design may =
be=20
flawed.  What is 'one go'? How many 'goes per second' do you expect to=20
sustain?

I meant ( basically the same, but the actual code is slightly different =
than
the following ):

--- code excerpt from NeoBoard ---

def getArticles(...):

... prepare....

articles =3D [article for article in board.ZopeFind( =
obj_metatypes=3D'NeoBoard
Article' , search_sub=3D1)]

# I use ZopeFind instead of objectValues since articles get nested

...sort articles...
...return articles...

---------------------------

Note that my python/zope experience is limited. I'm still learning :-)

Suppose the 'board' folderish object ( let's assume it's a BTreeFolder; =
if
it's present NeoBoard uses BTreeFolder2, otherwise Zope Folder ) holds =
about
10,000 articles.

Whenever a list of articles gets displayed, this statement should be
executed. Most RDBMS-based boards do the same with a select/limit combo =
like
the PHP NeoBoard does:

"SELECT article_field_list from board limit start_num, end_num"

Am I missing something? Is there a way in Zope to limit the returned =
results
as a limit clause would do in an SQL statement?=20

Even the Zope core source code seems to get every subobject from =
folderish
objects no matter how many there might be. That gave me the impression =
that
Zope doesn't give you that option. Correct me if I'm wrong.

With RDBMS, I can limit the numver of returned rows with a limit clause =
that
is constructed with a request variable, say, 'b_start'. In Zope, I have =
to
pull all the rows (objects) first and get a specified batch from them by
wrapping them into batches.

I didn't mean 'goes per second'. Just wanted to know how much slower =
this
one go of pulling too many objects from a folder would be in comparison =
to
RDBMS's select/limit combo.

> How slow would it be compared to an SQL "select * from=20
> heavily_populated_board limit 50" (supposing a pageful of article list =

> would spew out 50 articles )? ( I just assume it'd be slower than=20
> using MySQL. Correct me if I'm wrong. )

>>The DirectoryStorage overhead for this is tiny: nothing more than =
reading
100=20
>>files from some nested directories and some simple validation on each =
file

>>header. Not much different to FileStorage, which would have 50 seeks =
and
50=20
>>reads in one file. Both storages would transfer the same number of =
data=20
>>bytes.

>>The key performance issue is how fast your filesystem can traverse =
those=20
>>directories. Or for FileStorage, the cost of keeping that=20
>>directory/dictionary in memory. If you have tried FileStorage and it
lacked=20
>>raw speed, then I doubt DirectoryStorage will be faster.

I applologize for not being specific with the query. I didn't mean to
compare DS with FS. I already know DS could be slower than FS and guess
that's acceptable given its benefits.

I was just curious how slower NeoBoard might get if I rely solely on =
ZODB (
irregardless of its storage model ) than on RDBMS. In fact, PHP/MySQL
NeoBoard is pretty fast. Given the skins and stuff, Zope NeoBoard should =
be
slower. But I found (and am a bit worried) that pulling data from ZODB =
is
too much slower than selecting rows from MySQL. I'd have to re-design
NeoBoard to use MySQL as storage backend like I did in good ol' days.
bummer...

Bear with my ignorance^^ You have every right not to answer this stupid
question ;-)

Thanks for your time anyway.

Best Regards,
Wankyu Choi
---------------------------------------------------------------
      To the dedicated staff at NeoQuest, language is not a problem
          to be dealt with, but an art waiting to be performed.
  ---------------------------------------------------------------
  Wankyou Choi
  CEO/President
  NeoQuest Communications, Inc.
  3rd Floor, HMC Bldg., 730-14, Yoksam-dong, Kangnam-gu
  Seoul, Korea
  Tel: 82-2 - 501 - 7124 Fax: 82-2-501-7058
  Corporate Home: http://www.neoqst.com
  Personal  Home: http://www.zoper.net, http://www.neoboard.net
  e-mail:   wankyu@neoqst.com
--------------------------------------------------------------- =20