Hi gang, This version just fixes a bug with the folder contents tab on Windows. For some reason, on Windows 95 (and perhaps other platforms) the Python interpreter behaves differently enough that if you use a class attribute from a different module and put it in your own class, you may in fact be modifying its global context. It's probably a bug in Python. As a workaround, this version of BTreeFolders doesn't have the "details" tab anymore. http://www.zope.org/Members/hathawsh/BTreeFolder/BTreeFolder-0.1.1.tar.gz BTW has anyone done any semi-formal speed tests? It's rumored that a custom getattr in Python can cause a bit of a slowdown. We could write a custom getattr in C to solve the problem. But we don't want to do any premature optimization. Shane
Shane Hathaway wrote:
Hi gang,
This version just fixes a bug with the folder contents tab on Windows. For some reason, on Windows 95 (and perhaps other platforms) the Python interpreter behaves differently enough that if you use a class attribute from a different module and put it in your own class, you may in fact be modifying its global context. It's probably a bug in Python. As a workaround, this version of BTreeFolders doesn't have the "details" tab anymore.
http://www.zope.org/Members/hathawsh/BTreeFolder/BTreeFolder-0.1.1.tar.gz
BTW has anyone done any semi-formal speed tests? It's rumored that a custom getattr in Python can cause a bit of a slowdown. We could write a custom getattr in C to solve the problem. But we don't want to do any premature optimization.
Shane
I'll work on some tests. Kapil
Kapil Thangavelu wrote:
Shane Hathaway wrote:
BTW has anyone done any semi-formal speed tests? It's rumored that a custom getattr in Python can cause a bit of a slowdown. We could write a custom getattr in C to solve the problem. But we don't want to do any premature optimization.
Shane
I'll work on some tests.
Kapil
semi-formal test i ran some basic tests comparing a btree folder against a folder. i'm not sure about my testing methodology but here are the results. the test code is at the end of this email. the first pair of numbers is a timed creation test for size xxx of DTML Documents. the second pair of numbers is a timed test for len(objectIds()) the third pair of numbers is a timed test for attribute access with a call to int(dtmldoc.title_or_id()) # all the ids are stringed numbers my test box is running mandrake 7.1 with 512 M ram, dual proc celerons 550s (oc'd) some quick conclusions. the btree folder is slower on average for creation by anywhere from 15-25% andt btree's __getattr is on average almost twice as slow as the folder's getattr. order folder followed by btreefolder for size 100 0.0974419116974 :::0.126843929291 0.000378012657166 :::0.000354051589966 0.0101770162582 :::0.0186030864716 for size 100 0.0978569984436 :::0.125252962112 0.000376105308533 :::0.000353932380676 0.0101470947266 :::0.0185579061508 for size 500 0.52112698555 :::0.657492995262 0.00154399871826 :::0.00154197216034 0.0524510145187 :::0.0956329107285 for size 500 0.53660595417 :::0.714686989784 0.00154197216034 :::0.00157594680786 0.0532569885254 :::0.100463986397 for size 1000 1.11411702633 :::1.40103697777 0.00302505493164 :::0.00306105613708 0.105547070503 :::0.193196058273 for size 1000 1.12388503551 :::1.58762395382 0.00370097160339 :::0.00381696224213 0.106634020805 :::0.194367051125 for size 2000 2.60056698322 :::3.20675897598 0.007483959198 :::0.00741696357727 0.216197967529 :::0.396993994713 for size 2000 2.64964604378 :::3.26421093941 0.0074850320816 :::0.00761103630066 0.214147925377 :::0.40997505188 for size 5000 9.50434195995 :::11.0002789497 0.0184400081635 :::0.018413901329 0.533676981926 :::0.997251987457 just to be fair i reversed the positions and did a second set of runs with only 1 run per size as the results were consistent with the first set. for size 100 0.128319978714 :::0.10124707222 0.000413060188293 :::0.000383019447327 0.018816113472 :::0.0104240179062 for size 100 0.12813603878 :::0.101204037666 0.00040602684021 :::0.000380039215088 0.0188159942627 :::0.0105849504471 for size 500 0.681550979614 :::0.546862006187 0.00152194499969 :::0.00160706043243 0.0983099937439 :::0.0540620088577 for size 1000 1.47808301449 :::1.17385303974 0.00308799743652 :::0.00319290161133 0.197283029556 :::0.108732938766 for size 5000 11.1204429865 :::9.15097498894 0.015163064003 :::0.0150960683823 1.00954806805 :::0.535046935081 Despite this evidence. at size 5000, looking at the btreefolder contents in the management screen was near instaneous while the folder took almost a minute. based on the above evidence though that has more to do with the interface than the underlying speed of the folder methods. night. Kapil test code to follow. external method import time import os import Products.BTreeFolder def btest(self, REQUEST, MMMM=None): ''' ''' if not MMMM: MMMM=100 results = [] bf = Products.BTreeFolder.BTreeFolder.BTreeFolder() bf.id = 'btree_test_b' self._setObject(bf.id, bf) id = 'btree_test_f' rf = self.manage_addFolder(id) # creation test for completeness stime = time.time() for num in xrange(MMMM): self.btree_test_b.manage_addDTMLDocument(str(num)) etime = time.time() result = etime-stime #self.get_transaction().commit() stime = time.time() for num in xrange(MMMM): self.btree_test_f.manage_addDTMLDocument(str(num)) etime = time.time() results.append(result, etime-stime) # access test stime = time.time() len(self.btree_test_b.objectIds()) etime = time.time() result = etime-stime stime = time.time() len(self.btree_test_f.objectIds()) etime = time.time() results.append(result, etime-stime) # iteration test stime = time.time() for oid, item in self.btree_test_b.objectItems(): int(item.title_or_id()) etime = time.time() result=etime-stime stime = time.time() for oid, item in self.btree_test_f.objectItems(): int(item.title_or_id()) etime = time.time() results.append(result, etime-stime) return results
Kapil Thangavelu wrote:
semi-formal test
i ran some basic tests comparing a btree folder against a folder. i'm not sure about my testing methodology but here are the results. the test code is at the end of this email.
(snip)
some quick conclusions. the btree folder is slower on average for creation by anywhere from 15-25% andt btree's __getattr is on average almost twice as slow as the folder's getattr.
I must say thank you! Your results look very conclusive. It would appear, then, that storing the objects in a BTree actually slows down reads and writes considerably. And the only thing that needs to be improved about folders is the user interface. The problem with the current Folder UI, you see, is that each object is loaded from the database every time you look at the folder contents view. Additionally, browsers have trouble rendering large tables. BTreeFolder gets the contents listing from the _objects attribute and generates a simple list. It's just lucky that all platforms I know of can handle thousands of items in a simple list without any slowdown. This provided us the data we needed for improving folders. Thanks again! Shane
On Mon, 28 Aug 2000 20:41:05 -0700, Kapil Thangavelu <kthangavelu@earthlink.net> wrote:
i ran some basic tests comparing a btree folder against a folder. i'm not sure about my testing methodology but here are the results. the test code is at the end of this email.
Im not sure this benchmark is measuring the things that a BTreeFolder should excel at.... One advantage of a BTreeFolder is that you can access *one* if its contained items without having to load *all* of the others. All of your tests iterate through the whole content. Secondly, it should be possible to add two items simultaneously without causing a conflict, however your creation test has no concurrency. Toby Dickenson tdickenson@geminidataloggers.com
Oh, well, I'd bet this is a silly question. But I can not resist nor find answer by myself. Q: Why don't you BTree-fy all the subclasses of ObjectManager? Is there a reason that anyone but can a newbie see? LEE Kwan Soo. ps. I only hope to be able to make clear statement in English, but i am already too sleepy now. sorry. ----- Original Message ----- From: "Shane Hathaway" <shane@digicool.com> To: <zope-dev@zope.org> Sent: Monday, August 28, 2000 10:26 PM Subject: [Zope-dev] New version of BTreeFolder
Hi gang,
This version just fixes a bug with the folder contents tab on Windows. For some reason, on Windows 95 (and perhaps other platforms) the Python interpreter behaves differently enough that if you use a class attribute from a different module and put it in your own class, you may in fact be modifying its global context. It's probably a bug in Python. As a workaround, this version of BTreeFolders doesn't have the "details" tab anymore.
http://www.zope.org/Members/hathawsh/BTreeFolder/BTreeFolder-0.1.1.tar.gz
BTW has anyone done any semi-formal speed tests? It's rumored that a custom getattr in Python can cause a bit of a slowdown. We could write a custom getattr in C to solve the problem. But we don't want to do any premature optimization.
Shane
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
LEE Kwan Soo wrote:
Oh, well, I'd bet this is a silly question. But I can not resist nor find answer by myself.
Q: Why don't you BTree-fy all the subclasses of ObjectManager? Is there a reason that anyone but can a newbie see?
Obviously what BTreeFolder does should be available in standard folders. That is a long-term goal for Zope. BTreeFolder only exists to scope out the need for big folders as well as provide the capability in the short term. Shane
Shane Hathaway wrote:
LEE Kwan Soo wrote:
Oh, well, I'd bet this is a silly question. But I can not resist nor find answer by myself.
Q: Why don't you BTree-fy all the subclasses of ObjectManager? Is there a reason that anyone but can a newbie see?
Obviously what BTreeFolder does should be available in standard folders. That is a long-term goal for Zope. BTreeFolder only exists to scope out the need for big folders as well as provide the capability in the short term.
Any chance at a BTree Folder w/customizer suppor tin the near future? :) -- Do not meddle in the affairs of sysadmins, for they are easy to annoy, and have the root password.
Shane Hathaway wrote:
Bill Anderson wrote:
Any chance at a BTree Folder w/customizer support in the near future? :)
It should be quite easy to do. I vaguely recall Steve A or Steve S might have done it.
I want to include a BTreeFolder w/ Customizer support in the next release of DataSkinAddons. If anyone's done one, and you're happy for it to be included in DataSkinAddons, please let me know. Otherwise, I'll work on this later today. I've just produced a BTreeDataSkin: ---- BTreeDataSkin is a DataSkin that is an ObjectManager. The ObjectManager stores subobjects in a BTree that is directly in the object (not stored in an attribute provider). On __getattr__, subobject ids are searched before the datamanager is asked to provide attributes. ---- -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net
At 09:24 AM 8/29/00 +0100, Steve Alexander wrote:
Shane Hathaway wrote:
Bill Anderson wrote:
Any chance at a BTree Folder w/customizer support in the near future? :)
It should be quite easy to do. I vaguely recall Steve A or Steve S might have done it.
I want to include a BTreeFolder w/ Customizer support in the next release of DataSkinAddons.
If anyone's done one, and you're happy for it to be included in DataSkinAddons, please let me know.
Otherwise, I'll work on this later today.
I've just produced a BTreeDataSkin: ---- BTreeDataSkin is a DataSkin that is an ObjectManager. The ObjectManager stores subobjects in a BTree that is directly in the object (not stored in an attribute provider). On __getattr__, subobject ids are searched before the datamanager is asked to provide attributes. ----
If I may offer a suggestion... it would probably be a lot cleaner to implement this as an AttributeProvider, and use the dataskin's "persistent data slot" to store the b-tree. Otherwise, your b-tree contents can only be used with DataSkins which are not stored "virtually" in a Rack. If you use the slot API, it will work with any DataSkin. In other words, a BTreeAttributeProvider which simply stores attributes in a persistent BTree. Such a thing would be useful independent of the BTreeFolder user interface and could work with any DataSkin. Here's some code that implements a similar notion (using a dictionary in the DataSkin's storage slot, rather than a BTree). It's from the version of ZPatterns I'm currently working on. eaKey = 'ZPatterns.AttributeProviders', 'ExternalAttributes' class ExternalAttributeProvider(AttributeProvider): """ This attribute provider stores persistent attributes outside the object itself, in its persistent 'slot'. This allows persistent attributes to be used even with non-persistent DataSkins (e.g. those which are only virtually stored in a Rack. """ def _AttributeFor(self,client,name,default=None): return client._v_readableSlot.get(eaKey,{}).get(name,default) def _SetAttributeFor(self,client,name,value): """Set the attribute and return true if successful""" attrs = client._v_readableSlot.get(eaKey,{}); attrs[name]=value client._v_writeableSlot[eaKey] = attrs return 1 def _DelAttributeFor(self,client,name): """Delete the attribute and return true if successful""" if client._v_changedAttrs_[name] is NOT_FOUND: del client._v_changedAttrs_[name] attrs = client._v_readableSlot.get(eaKey,{}) if attrs.has_key(name): del attrs[name] client._v_writeableSlot[eaKey] = attrs return 1 # Class Metadata meta_type='Persistent External Attribute Provider' The code works by handling a persistent "slot". Currently, slots are used only by Persistent PropertySheet Providers to store propertysheets. The "_v_readableSlot"/"_v_writeableSlot" objects refer to a PersistentMapping which is associated with the DataSkin. By convention, the entries in the persistent mapping are keyed with a tuple that is the module name and purpose of the entry. This is why things are written to client._v_writableSlot[eaKey] -- eaKey is a constant that ensures that other providers needing to use the slot for persistent storage will not collide with it. You could probably take the code above, and, with some adaptation, produce a BTreeExternalAttributeProvider which used a BTree in place of a dictionary to store the attributes. You would need, of course, to use a different constant tuple to key the slot. _v_readableSlot always returns either a PersistentMapping, or an empty dictionary if the DataSkin's slot has never been written to. _v_writeableSlot always returns a PersistentMapping, ready to be written to. Once _v_writeableSlot has been accessed even once, the slot exists and _v_readableSlot will return the same PersistentMapping object as _v_writeableSlot. This rather odd-seeming setup is to prevent accidental changes to persistent storage when no changes have actually been made.
Bill Anderson wrote:
Any chance at a BTree Folder w/customizer suppor tin the near future? :)
I did some work towards this. However, I didn't continue. I realised that I could get the same effect by having a BTreeFolder inside a Customizer Folder. The only downside is a slightly longer URL for objects in the BTreeFolder. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net
Shane Hathaway wrote:
http://www.zope.org/Members/hathawsh/BTreeFolder/BTreeFolder-0.1.1.tar.gz
BTreeFolder doesn't do the from Globals import default__class_init__ default__class_init__(BTreeFolder) thing. Does that matter at all? As none of the classes it is derived from have a __init__ method, I guess not. Or, does default__class_init__ do anything else? -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net
Steve Alexander wrote:
BTreeFolder doesn't do the
from Globals import default__class_init__ default__class_init__(BTreeFolder)
thing.
Does that matter at all?
As none of the classes it is derived from have a __init__ method, I guess not. Or, does default__class_init__ do anything else?
It converts __ac_permissions__ into usable permissions declarations (__roles__ attributes.) default__class_init__ really ought to be there; it's only recently that Brian started saying we need to use it everywhere. I forgot. :-) Shane
participants (7)
-
Bill Anderson -
Kapil Thangavelu -
LEE Kwan Soo -
Phillip J. Eby -
Shane Hathaway -
Steve Alexander -
Toby Dickenson