[Zope-dev] zope.component test isolation (was: Zope test layers, pytest, and test isolation)

[Wolfgang Schnerring]

Hello Uli,

I've spent quite some time thinking (and partly coding) about the same
issues you mention (but didn't feel ready to talk about it here, yet),
so I'm glad that maybe we can start thinking about them together

I think your email addresses two quite different topics, so I'll split
my reply. First up: test support for zope.component. Second part:
about the concept of test layers.

* Uli Fouquet <uli at gnufix.de> [2011-03-24 01:05]:
> Right now we have a problem with pytest integration when it comes to
> ZCA setups [...] all tests share the same global ZCA registrations
> and changes to the registrations in one test will affect other tests
> run thereafter. We have a lack of test isolation.

Exactly. This issue has bitten me too in various places, and as far as
I know there are no solutions for it, yet.

What I envision to solve this issue is that test support for
zope.component should work the same way as with the ZODB. There, we
have a *stack* of Databases (DemoStorages, to be precise) that wrap
each other, where each one delegates reads downwards, but keeps writes
for itself. So you might have one database for the layer that provides
the baseline, and each test (in its setUp) gets its own database where
it can do whatever it wants, because it is thrown away in its
tearDown.

In principle, quite a few of the mechanics to do the same thing with
zope.component registries are already there (since a registry keeps a
list of base registries it will delegate to when something can not be
found in itself). And as Hanno and Godefroid mentioned, plone.testing
does something in this direction already. (And, it bears repeating,
in its core has no dependencies on Plone or Zope2.)

But as far as I see, there are issues that plone.testing does not
address:

1. I've been going over this stuff with my colleague Thomas Lotze, and
we realized that just squeezing in a new registry and bending its
bases to the previously active one is not enough for full isolation,
since this does not cover *deleting* registrations (one, you can only
delete the registration from the precise registry it was created in,
and two, in the just-bend-the-bases approach, once you delete a
registration, it's gone forever).

I think to provide full isolation, we need to make *copies*. And since
zope.component in general supports a chain of registries, we probably
need to make copies of each of them.

2. zope.component has two entry points, the global site registry and
the current registry (getGlobalSiteManager and getSiteManager).
The current registry can be anything, or more precisely, you can call
getSiteManager.sethook(callable) and provide a callable that returns
the current registry.

I think to provide test support for zope.component (i. e. generally,
at the "library level"), we need to support both entry points. The
global one is not hard, but the getSiteManager one gets nasty really
fast, because of course we have to rely on bending getSiteManager to
return the current "test registry" -- but anyone could at any time
call getSiteManager.sethook to change it! Which means we need to
intercept that and a) prevent our hook from being replaced and b)
inject the new registry into the test stack somehow.

As far as I understand, plone.testing sidesteps these issues by only
dealing with the global registry, and specially munging the two known
cases in the Zope world where getSiteManager is changed (zope.site and
five.something).

**

I'd like to know what people think about this plan.
Thomas and I have been over this quite a bit and think it's sound, not
overly complicated, and (after we did some experiments) definitely
doable. Please do point out stuff we missed. :-)

I'd very much like to put this functionality into zope.component
itself, which of course raises backwards compatibility issues galore,
but any code for this definitely isn't wasted since we can always
package it separately if we don't find a way to integrate it.

Thomas and I taken up implementing this, but we can't devote a lot of
time to it (about one session per week), so realistically I'm afraid I
guess it will take a few months until we have something of substance.
So if there are people who want to pitch in, that'd be great. I
definitely could write up a more detailed plan and maybe even
formulate smaller chunks so we could go at this with more people.

Wolfgang

-- 
Wolfgang Schnerring · ws at gocept.com
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1
Zope and Plone consulting and development