[Zope] ZCatalog speed?

[Stuart Woolford]

Well, I've been playing around with static DTML Documents and Zcatalogues from
my SQL database.

I use a set of dtml methods to generate static pages in a 2-level folder
heirachy from my SQL database, I use one parent method calling 3 sub-methods
(one to clear out the old structure and make a new one, one to get the sub-list
of folders, one to make the actual documents in the folders, this seems to
force Zope to commit a little more often than putting it all in one
document.. which is a good thing)

I have found the following:

on my 500MHz P-II with 196MB of memory it takes:

a - 22 minutes to create 8800 documents (smallish) in 1200 folders within zope,
not too fast :( but not exactly a user-interaction-limiting factor :)
b - too long to then try to do a search based add to a zcatalog,
ie: netscape times out after only around 8 minutes, and the search has not
finished!

BTW, Zopes python process and postgresql take about 50% of the CPU each, and
there is basically zero disk thrashing during this process (although zope does
get up around 50MB of memory use..)

Amy I trying to exceed the zcatalogs capabilities here? seems like a reasonable
ask to me, but without some reference it's hard to tell - I'm really wanting to
add 22000 larger documents, but that will have to wait till I find a faster way.

Research continuing :)

------------------------------------------------------------
Stuart Woolford, stuartw@newmail.net
Unix Consultant.
Software Developer.
Supra Club of New Zealand.
------------------------------------------------------------