[Zope] Problem: performance of retrieving properties from objects...

[Antwan Reijnen]

Hi all,

Due to a deadline in one week (yeah, I know, shame on stupid me!!!) and some problems I ran into, combined with my still too low Zope-experience (learning fast!), I could use some experience in datastorage matters in Zope... Of course, I would like to experiment myself as much as possible, but well, time is running out...

I am experimenting with adding and retrieving lots of properties on propertysheet objects, within ZClasses.

I find the performance of retrieving large sets of properties (say 400 props, strings, lines etcetera) in this way, way too slow for my application (on my slow pc (300 MHz) it takes 30 seconds). Also, while holding the number of retrieved properties constant, the retrieval-proces gets slower when the propertysheet contains more properties ;-( For the sake of the discussion, suppose I use propertyItems() on a propertysheet to retrieve the data.

So now I am forced to experiment further with another way to store my data. 

So maybe one or two experienced ZopeGuru's can shed a little light on the following possibilities (so I have to experiment a little less in my free (whoohaha) weekend, myself ;-):

1) Suppose I build a new python baseclass in which the bulk of my properties are stored. My slow ZClass object (actually a CMF object) uses this as a base class, and I write my own data-retrieval methods. How much will the increase in performance be? Would it be worth the while? Will I still be able to use the portal_catalog to catalog some interesting properties (I guess yes).

2) Suppose I have to switch to a database (there goes my weekend and free evenings next week). I want to retrieve the properties as one record then. I think the number of 400 props will decrease in this matter, too. I assume the database will retrieve these results a LOT faster. How much will the overhead be, however, to send such a dataset back to Zope...? How about using this database in combination with an existing catalog?

3) Suppose I build a CMF product entirely in python (may god help me), and I store all these properties as mentioned in (1), would the performance be higher than in case (1)?

Pfff, back to my application and coffee now. Any help and insight will be greatly appreciated... When I have time, I'll write a summary or a How-To on how to handle these kind of storage problems...

Greetings, TIA, Antwan.