Re: [Zope] Problem: performance of retrieving properties from objects...
Hi Lennart! Thanks for your prompt thoughts. Let me elaborate on some points:
The depends entirely on what the data retrival methods are...
I use two methods: one 'official' one: propertysheet.propertyItems(), and one I wrote myself, which fetches the properties loop-wise with propertysheet.getProperty(id). The time they require, is roughly the same...
2) Suppose I have to switch to a database (there goes my weekend and free evenings next week). I want to retrieve the properties as one record then. I think the number of 400 props will decrease in this matter, too.
Why would they decrease just because you move them to a relational database?
Quite af few properties are 'lines' and 'selection / multiple selection'. I guess when I use a rdb some of this zope-specific properties won't be nessecary.
I assume the database will retrieve these results a LOT faster.
That depends on the database... And it depends on why it is so slow. I think the fact that you have so many properties add to the slowness, since each of then has to be looked up.
Agree, but consider this: when I retrieve a fixed set of data, on propertysheets with varying number of properties (N), the fetchprocedure takes more time when N increases... Why should propertysheet.getProperty('aProperty') take longer on a larger propertysheet?
So a database would be faster here, since they have a fixed data structure. On the other hand, you might get the same effect by radically altering your data structure. Which brings us to the main question:
***Why on earth do you have 400 properties?***
I already drastically REDUCED it. I made 6 categories (propertysheets) which makes about 400 properties per propertysheet :-)) Each sheet contains 13 questions with lots of answerfields. I already told my boss that these kinds of large questionaries will not work in practice (BORING), but... you know, he is the boss ;-)
How about using this database in combination with an existing catalog?
It can be done, but is not necessary. Index the data in the database instead.
Ok. This way I have two different searches on the website, but that's acceptable.
3) Suppose I build a CMF product entirely in python (may god help me), and I store all these properties as mentioned in (1), would the performance be higher than in case (1)?
Python classes are supposedly faster than ZClasses (I don't use ZClasses, as I find them complicated, limited and cumbersome), so yes it would probably be faster, but I can't say how much
Ok, thanks again Lennart. If you have the time to enlighten me further, you are more than welcome!! Many greetings from the Netherlands, Antwan.
On Sat, 2002-04-06 at 02:31, Antwan Reijnen wrote:
Hi Lennart!
Thanks for your prompt thoughts. Let me elaborate on some points:
The depends entirely on what the data retrival methods are...
I use two methods: one 'official' one: propertysheet.propertyItems(), and one I wrote myself, which fetches the properties loop-wise with propertysheet.getProperty(id). The time they require, is roughly the same...
Is this done in a Script (Python)? or some other method in Zope? Are the results expected to change often? You may consider some caching. Wouldn't help the first fetch, but would, however, help follow-on requests. Also, do you need *all* of them at once? Does the need for some of them depend on certain variables? Of so, you may consider not getting the all at once, but rathe ronly those that you need right then.
2) Suppose I have to switch to a database (there goes my weekend and free evenings next week). I want to retrieve the properties as one record then. I think the number of 400 props will decrease in this matter, too.
Why would they decrease just because you move them to a relational database?
Quite af few properties are 'lines' and 'selection / multiple selection'. I guess when I use a rdb some of this zope-specific properties won't be nessecary.
What do you mean by "Zope-Specific"? ...
So a database would be faster here, since they have a fixed data structure. On the other hand, you might get the same effect by radically altering your data structure. Which brings us to the main question:
***Why on earth do you have 400 properties?***
Each sheet contains 13 questions with lots of answerfields. I already told my boss that these kinds of large questionaries will not work in practice (BORING), but... you know, he is the boss ;-)
So, would I be correct in presuming you are building a questionnaire, and using property sheets to store questions, and allowable answers (i.e. in a multiple choice question)?
From the limited info available, this sounds like a bad design choice, no offense intended. For something like this, I would consider making each questionnaire a container-like object and have the questions be sub-objects. Then, if you are storing the answers, perhaps an object that represents a completed questionnaire, that only stores the answers given. I believe there are also a few Zope Products that do this as well, you may consider looking into them.
-- Bill Anderson Linux in Boise Club http://www.libc.org Amateurs built the Ark, professionals built the Titanic. Amateurs build Linux, professionals build Windows(tm).
Antwan Reijnen writes:
... accessing properties with "getProperty" and "propertyItems".... Agree, but consider this: when I retrieve a fixed set of data, on propertysheets with varying number of properties (N), the fetchprocedure takes more time when N increases... Why should propertysheet.getProperty('aProperty') take longer on a larger propertysheet?
Currently, PropertySheets store their properties directly as attributes of the parent (i.e. the ZClass instance). Because of this, "getProperty" cannot relay on an "AttributeError" to see that "id" is really a property. It, therefore, calls "hasProperty" and then uses "getattr" on the parent. Properties have not been conceived as a mass storage concept. Therefore, the properties description is stored as a tuple. "hasProperty" iterates over the tuple and checks for the "id". This is linear in the size of the tuple. With current Zope, your most efficient way to access a property is "getattr" on the ZClass instance. Of course, this requires that you know what you access is really a property of the object. If not, you may get an acquired non-property. The PropertySheet implementers have been aware that the current storage mechanism (abusing the parent for storage) is a break in the PropertySheet's concept (different PropertySheets should provide separate namespaces; currently, they are couple by the parents namespace). Therefore, they documented that accessing properties via "getattr" from the parent may be removed in future versions. Personally, I do not expect this to happen, as it will break almost all ZClass applications. There may be new PropertySheets in Zope 3, that do it differently. But, for backward compatibility, the current sheets will almost surely continue to work as you see now. Dieter
Yes, there will most certainly be a totally different concept of properties in Zope 3. But you are right that changing them in Zope2 would be too painful and break too much legacy code 8^(. If you are looking for a way to store large quantities of mapped values, It would not be hard to implement a method using dictionaries or BTrees. You could even write it in such a way that it was compatible with the existing propertymanager/propertysheets interface and thus could use the same managment interface and methods. -Casey Dieter Maurer wrote:
Antwan Reijnen writes:
... accessing properties with "getProperty" and "propertyItems".... Agree, but consider this: when I retrieve a fixed set of data, on propertysheets with varying number of properties (N), the fetchprocedure takes more time when N increases... Why should propertysheet.getProperty('aProperty') take longer on a larger propertysheet?
Currently, PropertySheets store their properties directly as attributes of the parent (i.e. the ZClass instance). Because of this, "getProperty" cannot relay on an "AttributeError" to see that "id" is really a property. It, therefore, calls "hasProperty" and then uses "getattr" on the parent.
Properties have not been conceived as a mass storage concept. Therefore, the properties description is stored as a tuple. "hasProperty" iterates over the tuple and checks for the "id". This is linear in the size of the tuple.
[snip]
There may be new PropertySheets in Zope 3, that do it differently. But, for backward compatibility, the current sheets will almost surely continue to work as you see now.
Dieter
participants (4)
-
Antwan Reijnen -
Bill Anderson -
Casey Duncan -
Dieter Maurer