[ZODB-Dev] Nested OOBTrees?

[Jim Fulton]

Greg Ward wrote:
> 
> Hi all --
> 
> we're having some strange things going on in our database.  The
> background is that a broken web spider (Teleport Pro) recently went on a
> rampage on our web site, ignoring session cookies and /robots.txt; the
> result is that the number of active sessions in our live database
> exploded from its normal ~400-500 range to >7000.
> 
> Our database (the relevant area, at least) looks roughly like this:
> 
>   root['session_manager'] : MXSessionManager
>     .sessions             : OOBTrees { session_id:string -> session:MXSession }
> 
> (In English: one of our top-level objects right below the root is the
> session manager, an instance of MXSessionManager.  That instance as an
> attribute 'sessions', which is an OOBTree mapping session ID strings to
> instances of MXSession.)
> 
> Normally, root['session_manager'].sessions -- which henceforth I'll call
> 'sessions' -- has 400 or 500 sessions in it; we clean up old sessions
> nightly, so it doesn't grow too large.  The other night, it exploded to
> >7000 because of this idiotic web spider (and, err, our lack of defences
> against such spiders -- oops).
> 
> Now, the sessions OOBTree is acting strangely:
> 
> >> len(sessions)                  # OK (we have cleaned up some of the
> 3293                              # 7000 bogus sessions)
> >> len(sessions.keys())           # OK
> 3293
> >> list(sessions.keys())          # huh?!?
> []
> 
> (The values() and items() methods act similarly.)

That is rather odd. Is your BTree code up to date?
If so, I wouldn't mind getting a copy of the odd BTree.

...

> >> from ZODB.referencesf import referencesf
> >> (pkl, x) = storage.load(sessions._p_oid, None)
> >> oids = referencesf(pkl)
> 
> With me so far?  You would expect all of the objects referenced from an
> OOBTree to be OOBuckets, wouldn't you? 

No. They may be all buckets or all trees.

> (Well, *I* would, not from any
> deep understanding of the design, but simply because that's all I've
> ever seen.  But then I've never seen an OOBTree with thousands of
> objects before.)  But it isn't that way:
> 
> >> print connection[oids[0]]
> <OOBTree object at 0x814f4b8>
> >> print connection[oids[1]]
> <OOBTree object at 0x838c790>
> >> print connection[oids[2]]
> OOBucket([])
> 
> Wot the ... ?!?

Hm. I don't think that they should be a mix.
I don't think that there's an invarient that says that
they musn't be a mix, I just can't think off-hand how
they'd get to be a mix.
 
> So, two questions:
> 
>   * are OOBTrees "allowed" to include other OOBTrees?  or are they only
>     supposed to reference OOBuckets?
 
Yes. No.

>   * in either case, what's wrong with keys()/values()/items() on this
>     OOBTree?

I'd be happy to take a look at a particular BTree thar exhibits the
problem.

Jim