I've committed 2 patches to a "jinty-mem" branch of zope.interface. Together these patches reduce the startup memory use of my ZTK based application by 3%. Is that enough to justify their risk/complexity? https://mail.zope.org/pipermail/checkins/2010-November/052516.html https://mail.zope.org/pipermail/checkins/2010-November/052517.html If no-one replies, I'll assume that 3% is just not enough to be interesting and do nothing;) -- Brian Sutherland
Hi. On Mon, Nov 8, 2010 at 3:35 PM, Brian Sutherland <brian@vanguardistas.net> wrote:
I've committed 2 patches to a "jinty-mem" branch of zope.interface. Together these patches reduce the startup memory use of my ZTK based application by 3%. Is that enough to justify their risk/complexity?
https://mail.zope.org/pipermail/checkins/2010-November/052516.html https://mail.zope.org/pipermail/checkins/2010-November/052517.html
If no-one replies, I'll assume that 3% is just not enough to be interesting and do nothing;)
The changes looked all low-risk to me. I see no reason, why we shouldn't put them in. There's some quite complex applications built with zope.interface that might benefit even more from this. Hanno
On Mon, Nov 08, 2010 at 03:35:09PM +0100, Brian Sutherland wrote:
I've committed 2 patches to a "jinty-mem" branch of zope.interface. Together these patches reduce the startup memory use of my ZTK based application by 3%. Is that enough to justify their risk/complexity?
https://mail.zope.org/pipermail/checkins/2010-November/052516.html https://mail.zope.org/pipermail/checkins/2010-November/052517.html
If no-one replies, I'll assume that 3% is just not enough to be interesting and do nothing;)
It's so interesting I'm going to ask for even more numbers: how much is that 3% in kilobytes, or objects? Is there any measurable speed impact for application startup or test runs (faster/slower)? Otherwise I'm +1, assuming this doesn't somehow make the code 10% slower. Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development
On Tue, 2010-11-09 at 01:21 +0200, Marius Gedminas wrote:
On Mon, Nov 08, 2010 at 03:35:09PM +0100, Brian Sutherland wrote:
I've committed 2 patches to a "jinty-mem" branch of zope.interface. Together these patches reduce the startup memory use of my ZTK based application by 3%. Is that enough to justify their risk/complexity?
https://mail.zope.org/pipermail/checkins/2010-November/052516.html https://mail.zope.org/pipermail/checkins/2010-November/052517.html
If no-one replies, I'll assume that 3% is just not enough to be interesting and do nothing;)
I'm -1 on it for only 3%. ;-) - C
It's so interesting I'm going to ask for even more numbers: how much is that 3% in kilobytes, or objects? Is there any measurable speed impact for application startup or test runs (faster/slower)?
Otherwise I'm +1, assuming this doesn't somehow make the code 10% slower.
Marius Gedminas _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
On Tue, Nov 09, 2010 at 01:21:00AM +0200, Marius Gedminas wrote:
On Mon, Nov 08, 2010 at 03:35:09PM +0100, Brian Sutherland wrote:
I've committed 2 patches to a "jinty-mem" branch of zope.interface. Together these patches reduce the startup memory use of my ZTK based application by 3%. Is that enough to justify their risk/complexity?
https://mail.zope.org/pipermail/checkins/2010-November/052516.html https://mail.zope.org/pipermail/checkins/2010-November/052517.html
If no-one replies, I'll assume that 3% is just not enough to be interesting and do nothing;)
It's so interesting I'm going to ask for even more numbers: how much is that 3% in kilobytes, or objects?
Ok, you asked for it ;) The 3% was in kilobytes as measured using /bin/ps on OSX leopard with python2.6 from MacPorts. The app I measured is a ZTK 1.0 based web application. My application without the changes: Objects: 245614 RSS memory (kb): 59116 My application with the changes: Objects: 227147 RSS memory (kb): 57180 I also did a micro-benchmark (more below) which indicate that when creating an interface with 5 attributes and 5 methods you save 41% in kilobytes and 34% in objects.
Is there any measurable speed impact for application startup or test runs (faster/slower)?
My app doesn't startup measurably faster or slower. The ZTK tests seem to run a little faster, but the error is high on my laptop: Without patches: ./bin/test-ztk 71.93s ./bin/test-ztk 70.49s ./bin/test-ztk 70.31s With patches: ./bin/test-ztk 70.28s ./bin/test-ztk 70.00s ./bin/test-ztk 69.98s I also did some micro-benchmarking (script is attached). There were repeatable results which indicate that startup will be faster and runtime only probably faster: Without patches: Performance bench ----------------- one interface, 5 methods, 5 functions: 181.00 usec/pass five inheriting interfaces: 355.76 usec/pass one interface: 59.79 usec/pass query uninitialized tagged value: 1.26 usec/pass query initialized tagged value: 1.27 usec/pass Memory bench ------------ Memory before (kb): RSS VSZ 33684 109780 objects created: 150001 Memory after (kb): RSS VSZ 47856 124116 With patches: Performance bench ----------------- one interface, 5 methods, 5 functions: 165.97 usec/pass five inheriting interfaces: 341.22 usec/pass one interface: 53.03 usec/pass query uninitialized tagged value: 0.75 usec/pass query initialized tagged value: 1.37 usec/pass Memory bench ------------ Memory before (kb): RSS VSZ 33328 109524 objects created: 99001 Memory after (kb): RSS VSZ 41728 117972
Otherwise I'm +1, assuming this doesn't somehow make the code 10% slower.
Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
-- Brian Sutherland
Am 08.11.2010, 15:35 Uhr, schrieb Brian Sutherland <brian@vanguardistas.net>:
If no-one replies, I'll assume that 3% is just not enough to be interesting and do nothing;)
Hi Brian, thanks for sharing this but it looks to me like a micro-optimisation that isn't really worth going against best practice for - by using explicit name-mangling. - return self.__tagged_values.get(tag, default) + return getattr(self, '_Element__tagged_values', {}).get(tag, default) Any improvements higher up the stack (using slots, different Python version or compilers like LLVM) are likely to have a more significant effect. Might be worth timing a slots-based implementation. Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226
On Tue, Nov 09, 2010 at 03:49:38PM +0100, Charlie Clark wrote:
Am 08.11.2010, 15:35 Uhr, schrieb Brian Sutherland <brian@vanguardistas.net>:
If no-one replies, I'll assume that 3% is just not enough to be interesting and do nothing;)
Hi Brian,
thanks for sharing this but it looks to me like a micro-optimisation that isn't really worth going against best practice for - by using explicit name-mangling.
- return self.__tagged_values.get(tag, default) + return getattr(self, '_Element__tagged_values', {}).get(tag, default)
In response to Marius' comments I changed the patch a bit earlier to this: def queryTaggedValue(self, tag, default=None): """ Returns the value associated with 'tag'. """ tv = self.__tagged_values if tv is None: return default return tv.get(tag, default) Which is is faster as well as not doing the explicit name mangling you object to.
Any improvements higher up the stack (using slots, different Python version or compilers like LLVM) are likely to have a more significant effect.
That would be great, when it arrives. Actually, in my case, memory is the issue. So the only optimization that'll really help in a big way is if multiple python processes running similar code start sharing data. Some people have poked at that problem, but I havn't seen anything significant yet.
Might be worth timing a slots-based implementation.
I just tried to add slots to Element, Attribute and Method classes, but it there's too many tests that break to benchmark it running the ZTK tests. On the micro benchmarks, the only difference I see is a memory improvement which I think will not survive in practice because most
Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
-- Brian Sutherland
On Tue, Nov 09, 2010 at 05:04:11PM +0100, Brian Sutherland wrote:
Might be worth timing a slots-based implementation.
I just tried to add slots to Element, Attribute and Method classes, but it there's too many tests that break to benchmark it running the ZTK tests.
On the micro benchmarks, the only difference I see is a memory improvement which I think will not survive in practice because most
... ? Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development
On Wed, Nov 10, 2010 at 12:14:54AM +0200, Marius Gedminas wrote:
On Tue, Nov 09, 2010 at 05:04:11PM +0100, Brian Sutherland wrote:
Might be worth timing a slots-based implementation.
I just tried to add slots to Element, Attribute and Method classes, but it there's too many tests that break to benchmark it running the ZTK tests.
On the micro benchmarks, the only difference I see is a memory improvement which I think will not survive in practice because most
... ?
Er, somehow my thoughts didn't make it into text: attributes are zope.schema fields (i.e. subclasses of Attribute). To really make that work, you'd have to add __slots__ to most of the classes in zope.schema as well.
Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
-- Brian Sutherland
Am 10.11.2010, 01:40 Uhr, schrieb Brian Sutherland <brian@vanguardistas.net>:
On the micro benchmarks, the only difference I see is a memory improvement which I think will not survive in practice because most
... ? Er, somehow my thoughts didn't make it into text: attributes are zope.schema fields (i.e. subclasses of Attribute). To really make that work, you'd have to add __slots__ to most of the classes in zope.schema as well.
It might be purely academic but if this does significantly reduce memory use then it might be worth pursuing. Slots are supposed to be beneficial when you've got lots of objects around. Zope schema fields are pretty static anyway so I don't see a great penalty in having to add slots to them, although obviously a migration of a larger project would probably be a pain. Back to the micro-optimisation. At Jeff Rush's talk at PyCon this year he covered the penalties incurred by "." lookups so that might be something that could be investigated but might it also be possible to "memoize" the method calls? I keep forgetting when you can't do that. Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226
On Wed, Nov 10, 2010 at 12:07:00PM +0100, Charlie Clark wrote:
Am 10.11.2010, 01:40 Uhr, schrieb Brian Sutherland <brian@vanguardistas.net>:
On the micro benchmarks, the only difference I see is a memory improvement which I think will not survive in practice because most
... ? Er, somehow my thoughts didn't make it into text: attributes are zope.schema fields (i.e. subclasses of Attribute). To really make that work, you'd have to add __slots__ to most of the classes in zope.schema as well.
It might be purely academic but if this does significantly reduce memory use then it might be worth pursuing.
Turns out it's not that academic. I tried a bit harder and managed to apply __slots__ to the Element, Attribute and Method classes in zope.interface. That saved an additional 0.5 % of memory (bytes). I have a small patch to zope.schema that fixes the breakage there. It looks like there is an additional 0.5% available if zope.schema is refactored to use slots as well.
Slots are supposed to be beneficial when you've got lots of objects around. Zope schema fields are pretty static anyway so I don't see a great penalty in having to add slots to them, although obviously a migration of a larger project would probably be a pain.
Searching for __dict__ here: http://zope3.pov.lt/trac/browser/zope.schema/trunk/src/zope/schema/_bootstra... Will give you a good idea of the problems. Field.bind and ValidatedProperty both depend heavily on __dict__. For ValidatedProperty in particular I have no idea how that would work when using slots.
Back to the micro-optimisation. At Jeff Rush's talk at PyCon this year he covered the penalties incurred by "." lookups so that might be something that could be investigated but might it also be possible to "memoize" the method calls? I keep forgetting when you can't do that.
I had a look at http://en.wikipedia.org/wiki/Memoization, but I don't see how it could be applied in this case.
Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
-- Brian Sutherland
participants (5)
-
Brian Sutherland -
Charlie Clark -
Chris McDonough -
Hanno Schlichting -
Marius Gedminas