[Zope3-dev] Re: functional_tests

[Jim Fulton]

Garrett Smith wrote:
> Jim Fulton wrote:
> 
>> Garrett Smith wrote:
>>
>>> The files in the functional_tests look very intriguing. Are these 
>>> being used?
>>
>>
>>
>> No, They really should be deleted for now.
>>
>>> Is there a future for these types of tests in X3?
>>
>>
>>
>> Yes. See: http://cvs.zope.org/Packages/FunctionalTests/
>>
>> This is a function test framework that uses tcpwatch,
>> http://cvs.zope.org/Packages/tcpwatch/, to record web interactions
>> and play them back for testing.
>>
>> The main differences from ftests:
>>
>> - Use of tcpwatch to record requests.  It would be very nice
>>   to write something that coverted a tcpwatch log to ftests.
> 
> 
> I would think it desireable to hookup tcpwatch test files (not sure what 
> these look like) in the same way we hook up doctests. E.g.
> 
>   ftests.py
>   ---------
>   def test_suite():
>       return TcpWatchSuite(some.package, 'sometest.tcpw')
> 
> Is this what you mean by converting them to ftest?

Perhaps, though currently, with the functional tests, the
log files are tranfered to a different format and generally require
a lot of editing.  I suspect this could be automated more than
it has been. This is an opportunity for someone to excell. :)

> 
>> = Testing is totally external. The test process is outside
>>   the Zope process. This has advantages and disadvantages.
>>   They test the enitre system, while ftests bypass the server.
>>   On the other hand, these functional tests are far harder to run.
> 
> 
> My vote would be to run such tests on the temporary zodb/environment 
> created by the ftest framework.

Well, that seems attractive, depending on the amound of data
created.  In any case, this has not been automated AFAIK.

IN general, while I think this functional framework has a lot
of potential, it is a bot raw and currently requires a good bit
of effort to use.

 > I view 'external testing' as application
> specific.

I don't know what your point is.  All testing is application specific,
for some definition of the application.

In the future, I want to rearrange our automatd testing framework so
that it has a notion of layers.  When you define a test, you also define
the layer the software you are testing is built on.  This will imply
a baseline that is set up before running all the tests in a layer.
In this model, a test is testing always testing "application" code
wrt the layer.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org