Kapil Thangavelu wrote:

Shane Hathaway wrote:

...

BTW has anyone done any semi-formal speed tests? It's rumored that a custom getattr in Python can cause a bit of a slowdown. We could write a custom getattr in C to solve the problem. But we don't want to do any premature optimization.

Shane

I'll work on some tests.

Kapil

semi-formal test i ran some basic tests comparing a btree folder against a folder. i'm not sure about my testing methodology but here are the results. the test code is at the end of this email. the first pair of numbers is a timed creation test for size xxx of DTML Documents. the second pair of numbers is a timed test for len(objectIds()) the third pair of numbers is a timed test for attribute access with a call to int(dtmldoc.title_or_id()) # all the ids are stringed numbers my test box is running mandrake 7.1 with 512 M ram, dual proc celerons 550s (oc'd) some quick conclusions. the btree folder is slower on average for creation by anywhere from 15-25% andt btree's __getattr is on average almost twice as slow as the folder's getattr. order folder followed by btreefolder for size 100 0.0974419116974 :::0.126843929291 0.000378012657166 :::0.000354051589966 0.0101770162582 :::0.0186030864716 for size 100 0.0978569984436 :::0.125252962112 0.000376105308533 :::0.000353932380676 0.0101470947266 :::0.0185579061508 for size 500 0.52112698555 :::0.657492995262 0.00154399871826 :::0.00154197216034 0.0524510145187 :::0.0956329107285 for size 500 0.53660595417 :::0.714686989784 0.00154197216034 :::0.00157594680786 0.0532569885254 :::0.100463986397 for size 1000 1.11411702633 :::1.40103697777 0.00302505493164 :::0.00306105613708 0.105547070503 :::0.193196058273 for size 1000 1.12388503551 :::1.58762395382 0.00370097160339 :::0.00381696224213 0.106634020805 :::0.194367051125 for size 2000 2.60056698322 :::3.20675897598 0.007483959198 :::0.00741696357727 0.216197967529 :::0.396993994713 for size 2000 2.64964604378 :::3.26421093941 0.0074850320816 :::0.00761103630066 0.214147925377 :::0.40997505188 for size 5000 9.50434195995 :::11.0002789497 0.0184400081635 :::0.018413901329 0.533676981926 :::0.997251987457 just to be fair i reversed the positions and did a second set of runs with only 1 run per size as the results were consistent with the first set. for size 100 0.128319978714 :::0.10124707222 0.000413060188293 :::0.000383019447327 0.018816113472 :::0.0104240179062 for size 100 0.12813603878 :::0.101204037666 0.00040602684021 :::0.000380039215088 0.0188159942627 :::0.0105849504471 for size 500 0.681550979614 :::0.546862006187 0.00152194499969 :::0.00160706043243 0.0983099937439 :::0.0540620088577 for size 1000 1.47808301449 :::1.17385303974 0.00308799743652 :::0.00319290161133 0.197283029556 :::0.108732938766 for size 5000 11.1204429865 :::9.15097498894 0.015163064003 :::0.0150960683823 1.00954806805 :::0.535046935081 Despite this evidence. at size 5000, looking at the btreefolder contents in the management screen was near instaneous while the folder took almost a minute. based on the above evidence though that has more to do with the interface than the underlying speed of the folder methods. night. Kapil test code to follow. external method import time import os import Products.BTreeFolder def btest(self, REQUEST, MMMM=None): ''' ''' if not MMMM: MMMM=100 results = [] bf = Products.BTreeFolder.BTreeFolder.BTreeFolder() bf.id = 'btree_test_b' self._setObject(bf.id, bf) id = 'btree_test_f' rf = self.manage_addFolder(id) # creation test for completeness stime = time.time() for num in xrange(MMMM): self.btree_test_b.manage_addDTMLDocument(str(num)) etime = time.time() result = etime-stime #self.get_transaction().commit() stime = time.time() for num in xrange(MMMM): self.btree_test_f.manage_addDTMLDocument(str(num)) etime = time.time() results.append(result, etime-stime) # access test stime = time.time() len(self.btree_test_b.objectIds()) etime = time.time() result = etime-stime stime = time.time() len(self.btree_test_f.objectIds()) etime = time.time() results.append(result, etime-stime) # iteration test stime = time.time() for oid, item in self.btree_test_b.objectItems(): int(item.title_or_id()) etime = time.time() result=etime-stime stime = time.time() for oid, item in self.btree_test_f.objectItems(): int(item.title_or_id()) etime = time.time() results.append(result, etime-stime) return results

On Mon, 28 Aug 2000 20:41:05 -0700, Kapil Thangavelu <kthangavelu@earthlink.net> wrote:

i ran some basic tests comparing a btree folder against a folder. i'm not sure about my testing methodology but here are the results. the test code is at the end of this email.

Im not sure this benchmark is measuring the things that a BTreeFolder should excel at.... One advantage of a BTreeFolder is that you can access *one* if its contained items without having to load *all* of the others. All of your tests iterate through the whole content. Secondly, it should be possible to add two items simultaneously without causing a conflict, however your creation test has no concurrency. Toby Dickenson tdickenson@geminidataloggers.com

LEE Kwan Soo wrote:

Oh, well, I'd bet this is a silly question. But I can not resist nor find answer by myself.

Q: Why don't you BTree-fy all the subclasses of ObjectManager? Is there a reason that anyone but can a newbie see?

Obviously what BTreeFolder does should be available in standard folders. That is a long-term goal for Zope. BTreeFolder only exists to scope out the need for big folders as well as provide the capability in the short term. Shane

Bill Anderson wrote:

Any chance at a BTree Folder w/customizer support in the near future? :)

It should be quite easy to do. I vaguely recall Steve A or Steve S might have done it. Shane

At 09:24 AM 8/29/00 +0100, Steve Alexander wrote:

Shane Hathaway wrote:

...

Bill Anderson wrote:

...

Any chance at a BTree Folder w/customizer support in the near future? :)

It should be quite easy to do. I vaguely recall Steve A or Steve S might have done it.

I want to include a BTreeFolder w/ Customizer support in the next release of DataSkinAddons.

If anyone's done one, and you're happy for it to be included in DataSkinAddons, please let me know.

Otherwise, I'll work on this later today.

I've just produced a BTreeDataSkin: ---- BTreeDataSkin is a DataSkin that is an ObjectManager. The ObjectManager stores subobjects in a BTree that is directly in the object (not stored in an attribute provider). On __getattr__, subobject ids are searched before the datamanager is asked to provide attributes. ----

If I may offer a suggestion... it would probably be a lot cleaner to implement this as an AttributeProvider, and use the dataskin's "persistent data slot" to store the b-tree. Otherwise, your b-tree contents can only be used with DataSkins which are not stored "virtually" in a Rack. If you use the slot API, it will work with any DataSkin. In other words, a BTreeAttributeProvider which simply stores attributes in a persistent BTree. Such a thing would be useful independent of the BTreeFolder user interface and could work with any DataSkin. Here's some code that implements a similar notion (using a dictionary in the DataSkin's storage slot, rather than a BTree). It's from the version of ZPatterns I'm currently working on. eaKey = 'ZPatterns.AttributeProviders', 'ExternalAttributes' class ExternalAttributeProvider(AttributeProvider): """ This attribute provider stores persistent attributes outside the object itself, in its persistent 'slot'. This allows persistent attributes to be used even with non-persistent DataSkins (e.g. those which are only virtually stored in a Rack. """ def _AttributeFor(self,client,name,default=None): return client._v_readableSlot.get(eaKey,{}).get(name,default) def _SetAttributeFor(self,client,name,value): """Set the attribute and return true if successful""" attrs = client._v_readableSlot.get(eaKey,{}); attrs[name]=value client._v_writeableSlot[eaKey] = attrs return 1 def _DelAttributeFor(self,client,name): """Delete the attribute and return true if successful""" if client._v_changedAttrs_[name] is NOT_FOUND: del client._v_changedAttrs_[name] attrs = client._v_readableSlot.get(eaKey,{}) if attrs.has_key(name): del attrs[name] client._v_writeableSlot[eaKey] = attrs return 1 # Class Metadata meta_type='Persistent External Attribute Provider' The code works by handling a persistent "slot". Currently, slots are used only by Persistent PropertySheet Providers to store propertysheets. The "_v_readableSlot"/"_v_writeableSlot" objects refer to a PersistentMapping which is associated with the DataSkin. By convention, the entries in the persistent mapping are keyed with a tuple that is the module name and purpose of the entry. This is why things are written to client._v_writableSlot[eaKey] -- eaKey is a constant that ensures that other providers needing to use the slot for persistent storage will not collide with it. You could probably take the code above, and, with some adaptation, produce a BTreeExternalAttributeProvider which used a BTree in place of a dictionary to store the attributes. You would need, of course, to use a different constant tuple to key the slot. _v_readableSlot always returns either a PersistentMapping, or an empty dictionary if the DataSkin's slot has never been written to. _v_writeableSlot always returns a PersistentMapping, ready to be written to. Once _v_writeableSlot has been accessed even once, the slot exists and _v_readableSlot will return the same PersistentMapping object as _v_writeableSlot. This rather odd-seeming setup is to prevent accidental changes to persistent storage when no changes have actually been made.