- Zope-Dev - Zope lists

RE: [Zope-dev] BTreeFolder2.objectIds() - accessing _tree.keys() slow
by sean.upton＠uniontrib.com 14 Jan '06

14 Jan '06

> sean.upton(a)uniontrib.com wrote: > > I'm pretty sure this works. > > Ok, I get it now. I misread it the first time. > > > This returns the equivalent of running > > self.objectIds(spec=self._mt_index.keys()) on the current > > trunk/release code, which should be identical to > self._tree.keys(), but much, much faster. > > I'm still somewhat ignorant as to why self._tree.keys() is so slow > > with 100k-plus objects (waking up too many persistent objects?), > > I suspect the cost is in creating ghosts for all of the > persistent objects. > > No objections here--I like this patch. Thanks Shane - glad it makes sense. I don't have contributor rights - would you or anyone else be willing to gateway this diff for me and commit such changes? Thanks, Sean

1 0

Re: [ZODB-Dev] Re: [Zope-dev] Re: zLOG module deprecated
by Lennart Regebro 13 Jan '06

13 Jan '06

On 1/10/06, Andreas Jung <lists(a)andreas-jung.com> wrote: > This decision was made for Zope 2.8 (according to zLOG/__init__.py). How do you mean? As far as I can see, all the levels are still there in zLOG py. The question now is weather two of these levels should be removed, as I understand it because you don't understand why other people use them. I find that logic difficult to understand. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/

3 2

Re: [Zope-dev] Re: zLOG module deprecated
by Fred Drake 13 Jan '06

13 Jan '06

On 1/9/06, Andreas Jung <lists(a)andreas-jung.com> wrote: > ZODB defines these levels but I can not see any code in the ZODB package > that actually uses these levels. Nobody should be using the zLOG levels with the logging package, but rather use the logging package levels. So in the end, there's no need for Zope to be defining levels at all, only conventions for how the levels are used. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "There is no wealth but life." --John Ruskin

5 9

buildbot failure in Zope trunk 2.4 Linux zc-buildbot
by buildbot＠zope.com 13 Jan '06

13 Jan '06

The Buildbot has detected a failed build of Zope trunk 2.4 Linux zc-buildbot. Buildbot URL: http://buildbot.zope.org/ Build Reason: changes Build Source Stamp: 2751 Blamelist: andreasjung,efge,tim_one,ykomatsu BUILD FAILED: failed test sincerely, -The Buildbot

1 0

buildbot failure in Zope branches 2.9 2.4 Linux zc-buildbot
by buildbot＠zope.com 13 Jan '06

13 Jan '06

The Buildbot has detected a failed build of Zope branches 2.9 2.4 Linux zc-buildbot. Buildbot URL: http://buildbot.zope.org/ Build Reason: changes Build Source Stamp: 2750 Blamelist: andreasjung,anguenot,efge,mkerrin,oestermeier,tim_one,ykomatsu BUILD FAILED: failed test sincerely, -The Buildbot

1 0

Zope tests: 8 OK
by Zope tests summarizer 13 Jan '06

13 Jan '06

Summary of messages to the zope-tests list. Period Thu Jan 12 12:01:01 2006 UTC to Fri Jan 13 12:01:01 2006 UTC. There were 8 messages: 8 from Zope Unit Tests. Tests passed OK --------------- Subject: OK : Zope-2_6-branch Python-2.1.3 : Linux From: Zope Unit Tests Date: Thu Jan 12 21:01:50 EST 2006 URL: http://mail.zope.org/pipermail/zope-tests/2006-January/003994.html Subject: OK : Zope-2_6-branch Python-2.3.5 : Linux From: Zope Unit Tests Date: Thu Jan 12 21:03:20 EST 2006 URL: http://mail.zope.org/pipermail/zope-tests/2006-January/003995.html Subject: OK : Zope-2_7-branch Python-2.3.5 : Linux From: Zope Unit Tests Date: Thu Jan 12 21:04:50 EST 2006 URL: http://mail.zope.org/pipermail/zope-tests/2006-January/003996.html Subject: OK : Zope-2_7-branch Python-2.4.2 : Linux From: Zope Unit Tests Date: Thu Jan 12 21:06:20 EST 2006 URL: http://mail.zope.org/pipermail/zope-tests/2006-January/003997.html Subject: OK : Zope-2_8-branch Python-2.3.5 : Linux From: Zope Unit Tests Date: Thu Jan 12 21:07:50 EST 2006 URL: http://mail.zope.org/pipermail/zope-tests/2006-January/003998.html Subject: OK : Zope-2_8-branch Python-2.4.2 : Linux From: Zope Unit Tests Date: Thu Jan 12 21:09:22 EST 2006 URL: http://mail.zope.org/pipermail/zope-tests/2006-January/003999.html Subject: OK : Zope-2_9-branch Python-2.4.2 : Linux From: Zope Unit Tests Date: Thu Jan 12 21:10:52 EST 2006 URL: http://mail.zope.org/pipermail/zope-tests/2006-January/004000.html Subject: OK : Zope-trunk Python-2.4.2 : Linux From: Zope Unit Tests Date: Thu Jan 12 21:12:22 EST 2006 URL: http://mail.zope.org/pipermail/zope-tests/2006-January/004001.html

1 0

RE: [Zope-dev] BTreeFolder2.objectIds() - accessing _tree.keys() slow
by sean.upton＠uniontrib.com 13 Jan '06

13 Jan '06

> > Hacking objectIds() as follows (diff against trunk pasted inline) - > > gettting ids off of the meta type index for all used meta types - > > seems to make things much quicker. Two questions: > > Are you sure this actually works? _mt_index.keys() is > supposed to provide a list of all meta_types used in the > folder. To get the object ids from it, you'd need something > like this: > > ids = [] > for d in self._mt_index.values(): > ids.extend(d.keys()) > > The structure of _mt_index is documented in a comment: > > _mt_index = None # OOBTree: { meta_type -> OIBTree: { > id -> 1 } } I'm pretty sure this works. The in each OIBTree in _mt_index, the keys are ids of contained objects for all respective meta types in the folder - I use self._mt_index.keys() to list all meta_types and leverage your original code that runs when spec parameter has been passed gets run every time - loops through all meta_types does a union() of set and ids for each meta_type. This returns the equivalent of running self.objectIds(spec=self._mt_index.keys()) on the current trunk/release code, which should be identical to self._tree.keys(), but much, much faster. I'm still somewhat ignorant as to why self._tree.keys() is so slow with 100k-plus objects (waking up too many persistent objects?), but using the ids stored a few layers deep in the _mt_index seems a viable alternative with the same expected return result. With a bit more effort put in, the diff pasted below should be more complete and illustrate better: Index: BTreeFolder2.py =================================================================== --- BTreeFolder2.py (revision 41285) +++ BTreeFolder2.py (working copy) @@ -341,21 +341,22 @@ # Returns a list of subobject ids of the current object. # If 'spec' is specified, returns objects whose meta_type # matches 'spec'. - if spec is not None: - if isinstance(spec, StringType): - spec = [spec] - mti = self._mt_index - set = None - for meta_type in spec: - ids = mti.get(meta_type, None) - if ids is not None: - set = union(set, ids) - if set is None: - return () - else: - return set.keys() + + mti = self._mt_index + if spec is None: + spec = mti.keys() #all meta types + + if isinstance(spec, StringType): + spec = [spec] + set = None + for meta_type in spec: + ids = mti.get(meta_type, None) + if ids is not None: + set = union(set, ids) + if set is None: + return () else: - return self._tree.keys() + return set.keys() security.declareProtected(access_contents_information,

2 1

BTreeFolder2.objectIds() - accessing _tree.keys() slow
by sean.upton＠uniontrib.com 12 Jan '06

12 Jan '06

I have very large BTreeFolder2 (CMFMember via BaseBTreeFolder in Archetypes) - has about 260k items in _tree - objectIds() is painfully slow, as is self._tree.keys() - I've casually observed using the meta type index to get the object ids is many orders of magnitude faster. Hacking objectIds() as follows (diff against trunk pasted inline) - gettting ids off of the meta type index for all used meta types - seems to make things much quicker. Two questions: 1. Can this be considered safe in all cases (I'll code this behavior on a Subclass or in a monkey patch)? 2. If yes to #1, is there an appropriate place to lobby for this to be changed, perhaps in BTreeFolder2 product itself? Index: BTreeFolder2.py =================================================================== --- BTreeFolder2.py (revision 41285) +++ BTreeFolder2.py (working copy) @@ -341,10 +341,14 @@ # Returns a list of subobject ids of the current object. # If 'spec' is specified, returns objects whose meta_type # matches 'spec'. + + mti = self._mt_index + if spec is None: + spec = mti.keys() + if spec is not None: if isinstance(spec, StringType): spec = [spec] - mti = self._mt_index set = None for meta_type in spec: ids = mti.get(meta_type, None) Thanks, Sean +----------------------------------------------------------+ Sean Upton SignOnSanDiego.com Site Technology Supervisor The San Diego Union-Tribune 619.718.5241 sean.upton(a)signonsandiego.com 350 Camino De La Reina San Diego, CA 92108 Plone Powered! plone.org ++ python.org ++ zope.org +----------------------------------------------------------+

3 2

buildbot failure in Zope branches 2.9 2.4 Windows 2000 zc-bbwin2
by buildbot＠zope.com 12 Jan '06

12 Jan '06

The Buildbot has detected a failed build of Zope branches 2.9 2.4 Windows 2000 zc-bbwin2. Buildbot URL: http://buildbot.zope.org/ Build Reason: changes Build Source Stamp: 2735 Blamelist: andreasjung,dobe,jinty,mkerrin,oestermeier,poster,tim_one,yuppie BUILD FAILED: failed failed slave lost sincerely, -The Buildbot

1 0

buildbot failure in Zope trunk 2.4 Windows 2000 zc-bbwin6
by buildbot＠zope.com 12 Jan '06

12 Jan '06

The Buildbot has detected a failed build of Zope trunk 2.4 Windows 2000 zc-bbwin6. Buildbot URL: http://buildbot.zope.org/ Build Reason: changes Build Source Stamp: 2734 Blamelist: andreasjung,dobe,jinty,mkerrin,oestermeier,poster,tim_one,yuppie BUILD FAILED: failed failed slave lost sincerely, -The Buildbot

1 0