zope.sendmail Retry fixes and new state machine.
Hi! Just created a new branch for zope.sendmail. This is a result of the state machine analysis work I did on it after finding that it was creating problems on sending mail to our smart host... svn+ssh://grantma@svn.zope.org/repos/main/zope.sendmail/branches/grantma-retryfixes with lots of juicy new fixes including: - Restructured SMTP mailer and QueueProcessorThread so that all SMTP error logic is in the mailer. Clears the way for another mailer for /usr/sbin/sendmail command line can be used with QueueProcessorThread. - Added ability for QueueProcessorThread so that it can handle temporary failures in delivery to its smart host - ie administrator reconfiguring mailserver, mail server reboot/restart This was the main error path that was missing from the trunk code's state machine. - Formatted log messages in a consistent fashion so that they can be grepped out of z3.log - Added maildir message filename to log messages as message id - allows easy analysis/triage of mail message sending problems - Added optional cleaning of lock links to QueueProcessorThread so that messages can be resent immediately on Zope3 restart. - Added pollingInterval (ms), cleanLockLinks (boolean), and retryInterval (seconds) configure options to configure.zcml. Comments welcomed. I know it is a major portion of code and a bit of a restructure, but I have created unit tests for all my major changes. It includes all the stuff that was missing and that was needed for our production server. I imagine that all of this will prove to be really useful when it is added to the trunk zope.sendmail Cheers, Matthew Grant
--On 9. März 2008 21:11:21 +1300 Matthew Grant <grantma@anathoth.gen.nz> wrote:
Hi!
Just created a new branch for zope.sendmail.
This is a result of the state machine analysis work I did on it after finding that it was creating problems on sending mail to our smart host...
svn+ssh://grantma@svn.zope.org/repos/main/zope.sendmail/branches/grantma- retryfixes
with lots of juicy new fixes including:
- Restructured SMTP mailer and QueueProcessorThread so that all SMTP error logic is in the mailer. Clears the way for another mailer for /usr/sbin/sendmail command line can be used with QueueProcessorThread. - Added ability for QueueProcessorThread so that it can handle temporary failures in delivery to its smart host - ie administrator reconfiguring mailserver, mail server reboot/restart This was the main error path that was missing from the trunk code's state machine. - Formatted log messages in a consistent fashion so that they can be grepped out of z3.log - Added maildir message filename to log messages as message id - allows easy analysis/triage of mail message sending problems - Added optional cleaning of lock links to QueueProcessorThread so that messages can be resent immediately on Zope3 restart. - Added pollingInterval (ms), cleanLockLinks (boolean), and retryInterval (seconds) configure options to configure.zcml.
Comments welcomed. I know it is a major portion of code and a bit of a restructure, but I have created unit tests for all my major changes. It includes all the stuff that was missing and that was needed for our production server.
This sounds pretty impressing! Tnx, Andreas
These changes sound great. I just looked for your checkins in the checkins list. I didn't find them. If I recall correctly, this usually happens if the committer is not subscribed to the list and has not been blessed by an admin to let the messages through anyway. So, if I'm right, it would be cool if these happened: 1) someone with admin privs on the checkins list released Matthew's messages, and 2) either Matthew joined the checkins list or an admin blessed his messages so they could come through anyway. Gary On Mar 9, 2008, at 4:11 AM, Matthew Grant wrote:
Hi!
Just created a new branch for zope.sendmail.
This is a result of the state machine analysis work I did on it after finding that it was creating problems on sending mail to our smart host...
svn+ssh://grantma@svn.zope.org/repos/main/zope.sendmail/branches/ grantma-retryfixes
with lots of juicy new fixes including:
- Restructured SMTP mailer and QueueProcessorThread so that all SMTP error logic is in the mailer. Clears the way for another mailer for /usr/sbin/sendmail command line can be used with QueueProcessorThread. - Added ability for QueueProcessorThread so that it can handle temporary failures in delivery to its smart host - ie administrator reconfiguring mailserver, mail server reboot/restart This was the main error path that was missing from the trunk code's state machine. - Formatted log messages in a consistent fashion so that they can be grepped out of z3.log - Added maildir message filename to log messages as message id - allows easy analysis/triage of mail message sending problems - Added optional cleaning of lock links to QueueProcessorThread so that messages can be resent immediately on Zope3 restart. - Added pollingInterval (ms), cleanLockLinks (boolean), and retryInterval (seconds) configure options to configure.zcml.
Comments welcomed. I know it is a major portion of code and a bit of a restructure, but I have created unit tests for all my major changes. It includes all the stuff that was missing and that was needed for our production server.
I imagine that all of this will prove to be really useful when it is added to the trunk zope.sendmail
Cheers,
Matthew Grant _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Gary Poster wrote:
These changes sound great.
I just looked for your checkins in the checkins list. I didn't find them. If I recall correctly, this usually happens if the committer is not subscribed to the list and has not been blessed by an admin to let the messages through anyway.
So, if I'm right, it would be cool if these happened:
1) someone with admin privs on the checkins list released Matthew's messages, and 2) either Matthew joined the checkins list or an admin blessed his messages so they could come through anyway.
I just flushed a bunch of pending-good checkin messages from the admindb queue for the zope-checkins list. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH1F4K+gerLs4ltQ4RAqPUAJ9KLYVgwUnFsQlh9O+7gXyhaLNOagCdH5eo GE6Xg7P44syij9hmL3MwARs= =U+nr -----END PGP SIGNATURE-----
I did a five-minute skim of the checkin but hope to look a bit more tomorrow. Hopefully Marius, Benji, Albertas, or someone else who has actually done work on this package will take a look and chime in. I did have one somewhat trivial thought. I generally prefer durations and intervals expressed as datetime.timedeltas myself, because they convey their meaning without having to look it up docs (is that number value a number of seconds? milliseconds? minutes?). There might even be a zcml built in for schema field for that; I believe I remember that there is in ZConfig. Also, some variety of doctest would be nice. Even when a package is not using doctests, I add new tests as doctest unless there's a really good reason not to. In this case, it looks like you've made the code significantly more robust, which has added some probably necessary complexity. The code looks readable, but I recommend a maintainer-oriented overview/ introduction as a doctest, at the least. For instance, perhaps you could think about documentation about the rationale for the approach and about the dance that this code participates in (with the lock files and all the possible SMTP error conditions and the code's responses). Of course, even more friendly docs than that would be nice, but I'm only asking for what I myself tend to give, unfortunately. Gary
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Gary Poster wrote:
I did a five-minute skim of the checkin but hope to look a bit more tomorrow. Hopefully Marius, Benji, Albertas, or someone else who has actually done work on this package will take a look and chime in.
I did have one somewhat trivial thought. I generally prefer durations and intervals expressed as datetime.timedeltas myself, because they convey their meaning without having to look it up docs (is that number value a number of seconds? milliseconds? minutes?). There might even be a zcml built in for schema field for that; I believe I remember that there is in ZConfig.
Also, some variety of doctest would be nice. Even when a package is not using doctests, I add new tests as doctest unless there's a really good reason not to.
Becuase they make for poor unit tests? Using them to document the "mainline" use cases for an API is one thing: using them to do thorough coverage of edge cases is quite another. I find that for the latter, they fail *both* as documentation *and* as tests: their value as documentation drops as the amount of scaffoldiing rises, and the lack of isolation between tests sharply reduces their value for testing the corners. I realize I have said this before, but then others keep urging the "doctests everywhere" meme.
In this case, it looks like you've made the code significantly more robust, which has added some probably necessary complexity. The code looks readable, but I recommend a maintainer-oriented overview/ introduction as a doctest, at the least. For instance, perhaps you could think about documentation about the rationale for the approach and about the dance that this code participates in (with the lock files and all the possible SMTP error conditions and the code's responses). Of course, even more friendly docs than that would be nice, but I'm only asking for what I myself tend to give, unfortunately.
I would also value such documntation, but doubt that the scaffolding necessary to make the doctest part of the dance executable would make it any more valuable. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH1WRV+gerLs4ltQ4RAqHKAJ4lQgwkiCFIk1YpE6oon+xBbDT0sgCfa2M9 Uer6lWCkZh701UyspIzVso4= =5qeo -----END PGP SIGNATURE-----
Previously Tres Seaver wrote:
Gary Poster wrote:
I did a five-minute skim of the checkin but hope to look a bit more tomorrow. Hopefully Marius, Benji, Albertas, or someone else who has actually done work on this package will take a look and chime in.
I did have one somewhat trivial thought. I generally prefer durations and intervals expressed as datetime.timedeltas myself, because they convey their meaning without having to look it up docs (is that number value a number of seconds? milliseconds? minutes?). There might even be a zcml built in for schema field for that; I believe I remember that there is in ZConfig.
Also, some variety of doctest would be nice. Even when a package is not using doctests, I add new tests as doctest unless there's a really good reason not to.
Becuase they make for poor unit tests? Using them to document the "mainline" use cases for an API is one thing: using them to do thorough coverage of edge cases is quite another. I find that for the latter, they fail *both* as documentation *and* as tests: their value as documentation drops as the amount of scaffoldiing rises, and the lack of isolation between tests sharply reduces their value for testing the corners.
I realize I have said this before, but then others keep urging the "doctests everywhere" meme.
Indeed, and for that reason this can't be said enough. Doctests are useful to create testable documentation. They are not the right tool to create isolated, debuggable tests. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
Wichert Akkerman wrote:
Previously Tres Seaver wrote:
I realize I have said this before, but then others keep urging the "doctests everywhere" meme.
Indeed, and for that reason this can't be said enough. Doctests are useful to create testable documentation. They are not the right tool to create isolated, debuggable tests.
Not everyone agrees with this assertion. I for one vastly prefer both to read and write "unity" doctests over classic unit tests. And as "doctests everywhere" is the current status quo for just about the entire code base, I would expect a very convincing argument to be required to change. -- Benji York Senior Software Engineer Zope Corporation
Previously Benji York wrote:
Wichert Akkerman wrote:
Previously Tres Seaver wrote:
I realize I have said this before, but then others keep urging the "doctests everywhere" meme.
Indeed, and for that reason this can't be said enough. Doctests are useful to create testable documentation. They are not the right tool to create isolated, debuggable tests.
Not everyone agrees with this assertion. I for one vastly prefer both to read and write "unity" doctests over classic unit tests. And as "doctests everywhere" is the current status quo for just about the entire code base, I would expect a very convincing argument to be required to change.
The fact that something is popular does not necessarily mean it is the right thing :) Lack of isolation is a very convincing argument to me. Perhaps more personal taste but I also find python unittests to be much more readable. You don't suffer from mixing lots of test setup/teardown being repeated through the file. As Tres mentioned this is especially true when testing corner cases. Being able to debug tests by stepping over them with pdb is incredibly useful. With doctests that doesn't work. Being able to run a single test easily allows for quick testing and debugging. I can't tell the testrunner 'start running at line 52 but also include all the test setup magic from before, but skip the rest'. With unittests I can simple run zopectl test -s <module> -t <test function>. doctests hurt my productivity badly. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
On Mon, Mar 10, 2008 at 6:09 PM, Wichert Akkerman <wichert@wiggy.net> wrote:
The fact that something is popular does not necessarily mean it is the right thing :)
Lack of isolation is a very convincing argument to me.
Perhaps more personal taste but I also find python unittests to be much more readable. You don't suffer from mixing lots of test setup/teardown being repeated through the file. As Tres mentioned this is especially true when testing corner cases.
Being able to debug tests by stepping over them with pdb is incredibly useful. With doctests that doesn't work.
Being able to run a single test easily allows for quick testing and debugging. I can't tell the testrunner 'start running at line 52 but also include all the test setup magic from before, but skip the rest'. With unittests I can simple run zopectl test -s <module> -t <test function>.
doctests hurt my productivity badly.
I completely agree with Tres' and Wichert's statements on this. I only use doctests where they actually would make sense as documentation, the corner cases I always write as unit tests. The tools for dealing with pure Python code are so much more powerful than for python-embedded-in-text-with-prefixes, as well. -- Martijn Pieters
On Monday 10 March 2008, Martijn Pieters wrote:
I completely agree with Tres' and Wichert's statements on this. I only use doctests where they actually would make sense as documentation, the corner cases I always write as unit tests. The tools for dealing with pure Python code are so much more powerful than for python-embedded-in-text-with-prefixes, as well.
In my opinion it is as necessary to describe corner cases with words. In fact, corner cases often need even more documentation, because they are not obvious. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"
Stephan Richter wrote at 2008-3-10 22:37 -0500:
In my opinion it is as necessary to describe corner cases with words. In fact, corner cases often need even more documentation, because they are not obvious.
It depends. Many corner cases are obvious when one looks at the implementation. If they are not, I prefer words inline in the implementation (as a source code comment) over words separated in a test (also the test may specify in some way that it checks the specifiy corner case). -- Dieter
Wichert Akkerman wrote:
The fact that something is popular does not necessarily mean it is the right thing :)
Very true. It does, however, mean that you have to have a convincing argument to change popular opinion.
Lack of isolation is a very convincing argument to me.
This almost never bites me (if I'm understanding you correctly).
Perhaps more personal taste but I also find python unittests to be much more readable. You don't suffer from mixing lots of test setup/teardown being repeated through the file. As Tres mentioned this is especially true when testing corner cases.
Footnotes and good test structure alleviate the need for in-line setup/teardown.
Being able to debug tests by stepping over them with pdb is incredibly useful. With doctests that doesn't work.
As a heavy pdb user, this rarely bothers me, but seems to be more of an implementation issue than an argument that doctests are bad for unit tests.
Being able to run a single test easily allows for quick testing and debugging. I can't tell the testrunner 'start running at line 52 but also include all the test setup magic from before, but skip the rest'. With unittests I can simple run zopectl test -s <module> -t <test function>.
I don't see a huge advantage over bin/test -s <package> -t <doctest file name>
doctests hurt my productivity badly.
<shrug> I guess I'm the odd-man-out. -- Benji York Senior Software Engineer Zope Corporation
On Mar 10, 2008, at 1:32 PM, Benji York wrote:
doctests hurt my productivity badly.
<shrug> I guess I'm the odd-man-out.
Nope. You aren't. In fact, unittest hurt my productivity because they are almost always unfathomable. Jim -- Jim Fulton Zope Corporation
On Mon, Mar 10, 2008 at 06:09:50PM +0100, Wichert Akkerman wrote: [...the usual doctest vs unittest discussion ...]
Lack of isolation is a very convincing argument to me.
For me as well. Which is why I put my doctests into a traditional tests/test_foo.py module like this: # ... imports etc ... def doctest_FooClass_somemethod(): """Test FooClass.somemethod >>> foo = FooClass(ContextStub()) When you frob the groddle, by default you get zlotniks >>> foo.frob(GroddleStub()) '42 zlotniks' """ def doctest_FooClass_somemethod_special_case(): """Test FooClass.somemethod >>> foo = FooClass(ContextStub()) There's a very special corner case that happens when the system runs out of zlotniks >>> foo.frob(GroddleStub(zlotniks_required=1000)) Traceback (most recent call last): ... OutOfZlotniksError: 1000 zlotniks required, but only 50 are available """ def test_suite(): return doctest.DocTestSuite(setUp=setup.placelessSetUp, tearDown=setup.placelessTearDown, optionflags=doctest.NORMALIZE_WHITESPACE) I find that this style helps me write much more readable tests without sacrificing isolation. Instead I sacrifice pdb debuggability, that I don't value much (I find the pdb user interface horrible and prefer debugging with print statements). I also sacrifice test reusability -- you can't invoke one doctest from the middle of another one. (Footnotes don't count -- invoking a doctest footnote more than once is just *evil*, because you never know which invocation failed.). I find that for me personally, the advantages of test readability outweigh the disadvantages: * doctests without English test explaining each block of related assertions look naked. Unittests without comments look normal. As a result I feel the urge to explain my doctests when I never felt the urge to explain my unit tests. Don't know why that is, but that's how I feel. * The ability to nicely format data for presentation is extremely helpful. Compare >>> print_table([view.column_titles] + view.getTabularData()) +------------------+------------------+--------------+ | Month | Widgets produced | Widgets sold | +------------------+------------------+--------------+ | January | 42 | 17 | | February | 33 | 25 | +------------------+------------------+--------------+ with self.assertEquals(view.getTabularData(), [['January', 42, 17], ['February', 33, 25]]) Now compare the error messages when the test fails: Failed example: print_table([view.column_titles] + view.getTabularData()) Differences (ndiff with -expected +actual): +------------------+------------------+--------------+ | Month | Widgets produced | Widgets sold | +------------------+------------------+--------------+ - | January | 42 | 17 | ^ + | January | 43 | 17 | ^ | February | 33 | 25 | +------------------+------------------+--------------+ versus AssertionError: [['January', 42, 17], ['February', 33, 25]] != [['January', 43, 17], ['February', 33, 25]] I'm not saying that it's impossible to write a better table comparison function and use that instead of assertEquals. I'm saying that with doctests it's easier and feels more natural.
Perhaps more personal taste but I also find python unittests to be much more readable. You don't suffer from mixing lots of test setup/teardown being repeated through the file. As Tres mentioned this is especially true when testing corner cases.
My solution for repeating setup/teardown: * Put things common to all tests in the module (like placelessSetUp, setup.setUpAnnotations, registering your custom adapters needed for these tests) into global setUp/tearDown functions, and pass those to the DocTestSuite constructor. * Extract object creation into global reusable helper functions. Then tests look like >>> context = createFrombulator() >>> view = FrombulatorEditView(context, TestRequest()) and you don't have to care in every test that your Frombulator needs seventeen Wodgets in a nested tree of Grompets for correct functioning. These helper functions are parametrizable, so that if I find out that I need a very custom wodget in a particular test, I can do >>> my_wodget = Wodget('foo', 'bar') >>> my_wodget.someSetUpOnlyUsedInThisTest(42, -17j) >>> context = createFrombulator(main_wodget=my_wodget) ...
Being able to debug tests by stepping over them with pdb is incredibly useful. With doctests that doesn't work.
Right. In this case doctests aren't the right choice for you.
Being able to run a single test easily allows for quick testing and debugging. I can't tell the testrunner 'start running at line 52 but also include all the test setup magic from before, but skip the rest'.
This is one of the reasons why I prefer doctest modules to long documentation-and-all-tests-rolled-into-one .txt files, despite best efforts of Stephan Richter to convince me otherwise. ;-)
With unittests I can simple run zopectl test -s <module> -t <test function>.
Heh. I've got a vim macro that (1) finds the dotted name of the module I'm editing and (2) finds the nearest function definition above the cursor and (3) constructs a command line to run bin/test -s subdir -m package.module -t testnamee Yummy. Sadly, vim is completely incapable of sanely executing a long-running (which I define as "longer than 2 seconds") shell process in the background, so I have to alt-tab to a terminal window and paste the constructed command line that my vim macro puts into the clipboard.
doctests hurt my productivity badly.
I'm happy to say that's not the case for me. Now, *functional* tests do hurt my productivity sometimes. They also happen to be long-running slow doctests that compare multi-thousand-line HTML pages to ad-hoc patterns built with the doctest.ELLIPSIS feature. Pain. A recent decision to bite the bullet and introduce a dependency on lxml so that we could compare just the interesting snippets of HTML, extracted with xpath queries, helps somewhat. The abysmal rate of ~6 zope.testbrowser requests per second on a modern 1.8 GHz Core 2 Duo machine doesn't, but that's a different topic. Marius Gedminas -- I code in vi because I don't want to learn another OS. :) -- Robert Love
Benji York wrote at 2008-3-10 13:03 -0400:
Wichert Akkerman wrote:
Indeed, and for that reason this can't be said enough. Doctests are useful to create testable documentation. They are not the right tool to create isolated, debuggable tests.
Not everyone agrees with this assertion. I for one vastly prefer both to read and write "unity" doctests over classic unit tests. And as "doctests everywhere" is the current status quo for just about the entire code base, I would expect a very convincing argument to be required to change.
Looks at the ZODB tests. They make heavy use of multiple inheritance to compose test classes out of component test classes and use the same tests e.g. for "FileStorage" as well as for "ZEO" tests. -- Dieter
Dieter Maurer wrote:
Looks at the ZODB tests. They make heavy use of multiple inheritance to compose test classes out of component test classes
Good point; the ZODB tests make a great argument for using doctest over unittest.
and use the same tests e.g. for "FileStorage" as well as for "ZEO" tests.
I've done similar things with doctests. It's a great idea that transcends any particular testing tool. -- Benji York Senior Software Engineer Zope Corporation
Benji York wrote at 2008-3-11 09:03 -0400:
Dieter Maurer wrote:
Looks at the ZODB tests. They make heavy use of multiple inheritance to compose test classes out of component test classes
Good point; the ZODB tests make a great argument for using doctest over unittest.
After you switched to "doctest" for the ZODB tests I will check whether this badly effects the quality of the the test suite as measured by the number of not catched bugs ;-) -- Dieter
On Mar 10, 2008, at 17:57 , Wichert Akkerman wrote:
Previously Tres Seaver wrote:
Gary Poster wrote:
Also, some variety of doctest would be nice. Even when a package is not using doctests, I add new tests as doctest unless there's a really good reason not to.
Becuase they make for poor unit tests? Using them to document the "mainline" use cases for an API is one thing: using them to do thorough coverage of edge cases is quite another. I find that for the latter, they fail *both* as documentation *and* as tests: their value as documentation drops as the amount of scaffoldiing rises, and the lack of isolation between tests sharply reduces their value for testing the corners.
I realize I have said this before, but then others keep urging the "doctests everywhere" meme.
Indeed, and for that reason this can't be said enough. Doctests are useful to create testable documentation. They are not the right tool to create isolated, debuggable tests.
Amen to that. Doctests are a useful testing tool, but no single tool can fit every situation. jens
On Monday 10 March 2008, Wichert Akkerman wrote:
Indeed, and for that reason this can't be said enough. Doctests are useful to create testable documentation. They are not the right tool to create isolated, debuggable tests.
Huh? I totally disagree. If you cannot explain your software in words, you do not understand it. As there are different kinds of programmatic tests, you can easily write different levels of doctests that have different audiences. I do this all the time. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"
--On 10. März 2008 22:33:13 -0500 Stephan Richter <srichter@cosmos.phy.tufts.edu> wrote:
On Monday 10 March 2008, Wichert Akkerman wrote:
Indeed, and for that reason this can't be said enough. Doctests are useful to create testable documentation. They are not the right tool to create isolated, debuggable tests.
Huh? I totally disagree. If you cannot explain your software in words, you do not understand it. As there are different kinds of programmatic tests, you can easily write different levels of doctests that have different audiences.
This sounds like writing doctests just for sake of having doctests for all and everything. I completely disagree with that. In complex algorithms edgecases are often only of interest for the person implementing the code in order for having a safety belt. For writing good doctests you have to be a good (English) writer at some point. Doctests come into the game if you want to tell other people about the usage of your module. It's basically not of interest for telling them all edgecases. Addressing edgecase is unittests is basically good enough for me. They sometime require more code to test and put this into a doctest is often just overhead. Andreas
Hi Andreas. I think your response gets to the heart of the issue. For software to be useful, it is often more important for folks other than the author to understand it. This can only occur with communication. Sometimes it is the understanding of edgecases in particular, that gets lost over time. I find generally that I cannot keep in memory all of the details of the code I write. Several months afterwards without my own documentation, these details can be lost. So it is just important if I need to come back to it myself as for anyone else. If the test is worth writing, it is not difficult to add a small amount of text to communicate about it. If the software is worth writing in the first place, I consider the code incomplete without doctests. For community projects, it is often the case that it is not the author maintaining the code in the future. This only strengthens the argument for doctests and the reason this has become the standard for zope. Regards, David Andreas Jung wrote:
usage of your module. It's basically not of interest for telling them all edgecases. Addressing edgecase is unittests is basically good enough for me. They sometime require more code to test and put this into a doctest is often just overhead.
Andreas
------------------------------------------------------------------------
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
--On 11. März 2008 09:42:11 -0300 David Pratt <fairwinds@eastlink.ca> wrote:
Hi Andreas. I think your response gets to the heart of the issue. For software to be useful, it is often more important for folks other than the author to understand it. This can only occur with communication. Sometimes it is the understanding of edgecases in particular, that gets lost over time. I find generally that I cannot keep in memory all of the details of the code I write. Several months afterwards without my own documentation, these details can be lost. So it is just important if I need to come back to it myself as for anyone else.
You can also document your edgecase in the same way as comments within your unittests. It matters that you have tests for the related edgecases. It's up to every developer to decide which test type is best for the individual case. If you're good in story telling (I am not so good), go with doctest but there can not be law that everything must be in doctests.
If the test is worth writing, it is not difficult to add a small amount of text to communicate about it. If the software is worth writing in the first place, I consider the code incomplete without doctests. For community projects, it is often the case that it is not the author maintaining the code in the future. This only strengthens the argument for doctests and the reason this has become the standard for zope.
Agreed. But unittests are for certain tests more suitable. E.g. if you need a complex and bigger fixture or something like that. Andreas
On Mar 11, 2008, at 9:46 AM, Andreas Jung wrote: ...
You can also document your edgecase in the same way as comments within your unittests.
This is true in theory, but in practice, it almost never happens. There is no guarantee that doctests will contain useful text, but they encourage text which has a greater chance of being useful than no text at all.
Agreed. But unittests are for certain tests more suitable. E.g. if you need a complex and bigger fixture or something like that.
Unit testst don't make it any easier or harder to write test fixtures. They *do* encourage the use of inheritance to write text fixtures, which tends, in my experience to make them harder to understand. Jim -- Jim Fulton Zope Corporation
Jim Fulton wrote at 2008-3-11 09:58 -0400:
On Mar 11, 2008, at 9:46 AM, Andreas Jung wrote: ...
You can also document your edgecase in the same way as comments within your unittests.
This is true in theory, but in practice, it almost never happens.
There is no guarantee that doctests will contain useful text, but they encourage text which has a greater chance of being useful than no text at all.
"DocTests" essentially provide examples. I find documentation that only consists of examples very bad documentation -- not concise, at the wrong abstraction level, expecting me to induce the concepts from the examples rather than to describe the concepts and *then* (maybe) make some examples. Faced with the Zope 3/inspect incompatibility ("https://bugs.launchpad.net/zope3/+bug/181371") I have hit such a bad documentation: I have two classes "C1" and "C2", "C2" inherits from "C1". "C1" magically gets an attribute "a", but "C2" does not. Moreover, "C2.a" raises an AttributeError. There is a DocTest, that shows me that "C2.a" should indeed raise an "AttributeError". Thus, I can see that the author had something in mind when he implemented this counter intuitive behaviour. But, there is nothing that tells me what this may have been and why "C1" is treated differently from "C2". Documentation and tests have different purposes: Documentation tells relevant facts to the reader. It should be on an adequate abstraction level. It should clearly describe the concepts and include examples at places where this is helpful to unterstand the concepts. It should not consist of examples only and leave the concepts undescribed. The primary purpose of tests it to ensure that there are few undetected bugs. Some tests need to go to the same detail level as the implementation itself in order to probe for bugs introduced by implementation edgecases -- a level too detailed for a good documentation (apart from source code documentation itself). If a good test coveradge it obtained by performing the same tests over and over again on different data sets, representing relevant uses cases, the better. I do not need and I do not want essentially the same documentation for each of these sets. The ZODB test suite makes very good use of this principle, executing e.g. the "ConflictResolution" tests on both "FileStorage" and "ZEOStorage". The documentation is concise and clear: class FileStorageTests(..., ConflictResolutionTests, ...): ... By the way, a good documentation uses the same principle. It simply states "a 'FileStorage' supports conflict resolution (see XXXX for a description of the 'conflict resolution concept)" and does not show examples for conflict resolution for each an every conflict resolving storage. -- Dieter
David Pratt wrote at 2008-3-11 09:42 -0300:
I think your response gets to the heart of the issue. For software to be useful, it is often more important for folks other than the author to understand it. This can only occur with communication. Sometimes it is the understanding of edgecases in particular, that gets lost over time.
The are different understanding levels: e.g. the level of understanding needed by the user of a piece of software and the level of understanding needed by its maintainer. I use e.g. "gcc". To do this, I need a basic understanding of the programming language, the compilation process, the "gcc" options and arguments and how they influence the compilation and build process. I need nothing to know about edge cases in the compiler implementation. I also do not need doctests -- not even those that provide examples for the various options (think e.g. of doctests for the options "-m64", "-Wunused-label", "-O3" -- all of them are better described by words (only) rather than doctests). Would I be a "gcc" maintainer, I would need a very different understanding. Tests would be vital (though not necessarily doctests). However, more important would be a good understanding of compilation concepts and how they have been implemented in the "gcc" implementation. I find another point essential: communication is not limited to tests. I find tests (examples) a bad communication vehicle -- see the "gcc" examples above. A clear concept description and an outline how the concepts are used in the implementation, well chosen names, good source code comments at difficult places are additional communication means -- often better ones as doctests....
... If the test is worth writing, it is not difficult to add a small amount of text to communicate about it. If the software is worth writing in the first place, I consider the code incomplete without doctests. For community projects, it is often the case that it is not the author maintaining the code in the future. This only strengthens the argument for doctests and the reason this has become the standard for zope.
There are other means beside "DocTest"s for communication and documentation -- often better ones.... -- Dieter
Andreas Jung wrote at 2008-3-11 06:43 +0100:
... This sounds like writing doctests just for sake of having doctests for all and everything. I completely disagree with that. In complex algorithms edgecases are often only of interest for the person implementing the code in order for having a safety belt. For writing good doctests you have to be a good (English) writer at some point. Doctests come into the game if you want to tell other people about the usage of your module. It's basically not of interest for telling them all edgecases. Addressing edgecase is unittests is basically good enough for me. They sometime require more code to test and put this into a doctest is often just overhead.
+1 -- Dieter
On Mar 10, 2008, at 12:39 PM, Tres Seaver wrote:
Gary Poster wrote:
[...] Doctest/unittest holy war: bah. I've heard the arguments, I don't see either side as perfect, and I've made my choice. I'm very fine with others having different opinions and choices. As Benji said, the doctest choice has been made for the zope.* tree, so, until the current choice changes, it's reasonable to encourage doctests for zope.sendmail. If folks want to agitate for more unit tests in zope.*, have at it, but that doesn't change where we are now. BTW, Wichert, trying to be helpful: if you do have to deal with someone else's doctests, you can in fact use pdb. Just put the set_trace() on the same line as the command you want to step into. You can't step into subsequent doctest lines, but that does not tend to cause me too many problems, IME. Maybe that's your complaint, though. (To be clear, Wichert, I grant the validity of other points you and Tres and others made. There are counter-arguments, and matters of taste. Other folks can argue about this, not me.)
In this case, it looks like you've made the code significantly more robust, which has added some probably necessary complexity. The code looks readable, but I recommend a maintainer-oriented overview/ introduction as a doctest, at the least. For instance, perhaps you could think about documentation about the rationale for the approach and about the dance that this code participates in (with the lock files and all the possible SMTP error conditions and the code's responses). Of course, even more friendly docs than that would be nice, but I'm only asking for what I myself tend to give, unfortunately.
I would also value such documntation, but doubt that the scaffolding necessary to make the doctest part of the dance executable would make it any more valuable.
Here's one disagreement I'll bother to stand up for. Making documentation testable does add value to technical docs like this, in that it encourages it to stay up-to-date with refactorings. It is certainly not perfect--the examples can be tested but not the exposition--but saying that it does not add value in this case seems extreme to me. Perhaps your argument, Tres, is that the effort outweighs the cost in this particular case. That's a more reasonable argument to me. Maybe the scaffolding will be arduous. However, I would expect that some approaches to the doctest scaffolding would in fact mirror the set up needed for the unit tests, and I'd encourage Matthew to give it a try before immediately giving up. Gary
Gary Poster wrote:
Perhaps your argument, Tres, is that the effort outweighs the cost in this particular case. That's a more reasonable argument to me. Maybe the scaffolding will be arduous.
I can certainly get on board with that argument.. I, myself, have added small unit tests to zope.sendmail. At the time I contemplated converting the tests to doctest, but it looked too painful because I didn't understand the meaning behind the assertions being made in the current tests. -- Benji York Senior Software Engineer Zope Corporation
I hate to add more to this discussion, but I think some folks might have some miss-conceptions about doctests: On Mar 10, 2008, at 12:39 PM, Tres Seaver wrote: ...
Becuase they make for poor unit tests? Using them to document the "mainline" use cases for an API is one thing: using them to do thorough coverage of edge cases is quite another. I find that for the latter, they fail *both* as documentation *and* as tests: their value as documentation drops as the amount of scaffoldiing rises, and the lack of isolation between tests sharply reduces their value for testing the corners.
It is certainly straightforward enough to create isolated doctests. For edge cases, I do typically create separate isolated short doctests that deal just with that case. The assertion that doctests don't allow isolation is simply not correct. I'm not sure what scaffolding you're referring to. Do you mean test set up? Or the tools for normalizing output? For test set up, I don't think there's any difference. I can sympathize with the skepticism with normalizing output. Sometimes, I think it might be better to write documented tests rather than executable documentation and use assertion of a more traditional nature, if the test is still readable, although I suspect that it will often be easier to normalize output rather than make lots of assertions about the output. Jim -- Jim Fulton Zope Corporation
On Tue, Mar 11, 2008 at 09:25:59AM -0400, Jim Fulton wrote:
It is certainly straightforward enough to create isolated doctests. For edge cases, I do typically create separate isolated short doctests that deal just with that case. The assertion that doctests don't allow isolation is simply not correct.
Can you think of a package offhand that I might look at to see a good example of this? I've got a lot of overgrown, brittle doctests at work that need gradual reorganization, and I'd love to see some good role models. -- Paul Winkler http://www.slinkp.com
On Mar 13, 2008, at 12:11 AM, Paul Winkler wrote:
On Tue, Mar 11, 2008 at 09:25:59AM -0400, Jim Fulton wrote:
It is certainly straightforward enough to create isolated doctests. For edge cases, I do typically create separate isolated short doctests that deal just with that case. The assertion that doctests don't allow isolation is simply not correct.
Can you think of a package offhand that I might look at to see a good example of this?
zc.buildout is one.
I've got a lot of overgrown, brittle doctests at work that need gradual reorganization, and I'd love to see some good role models.
Hopefully, zc.buildout isn't too bad. Some common doctest traps that I fall into: - Making test files too long -- even for mainline tests. - Writing test files that aren't good documentation. I've been trying out a new practice of creating ".test" files rather than ".txt" files when I don't plan to spend much effort on documentation. I still like to try to tell a story, even if the story is aimed at the implementor. I also don't mind trading off some isolation for flow. I do still prefer to put edge cases into their own tests. Jim -- Jim Fulton Zope Corporation
On Thursday 13 March 2008, Paul Winkler wrote:
On Tue, Mar 11, 2008 at 09:25:59AM -0400, Jim Fulton wrote:
It is certainly straightforward enough to create isolated doctests. For edge cases, I do typically create separate isolated short doctests that deal just with that case. The assertion that doctests don't allow isolation is simply not correct.
Can you think of a package offhand that I might look at to see a good example of this?
I am very happy about my tests in z3c.form. They demonstrate tests with different audiences in mind. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"
Stephan Richter wrote:
I am very happy about my tests in z3c.form. They demonstrate tests with different audiences in mind.
Rightly so; they're excellent and make transition away from formlib easy. \malthe
On Tue, Mar 11, 2008 at 09:25:59AM -0400, Jim Fulton wrote:
I'm not sure what scaffolding you're referring to. Do you mean test set up? Or the tools for normalizing output? For test set up, I don't think there's any difference. I can sympathize with the skepticism with normalizing output. Sometimes, I think it might be better to write documented tests rather than executable documentation and use assertion of a more traditional nature, if the test is still readable, although I suspect that it will often be easier to normalize output rather than make lots of assertions about the output.
My personal experience with doctest output normalizers (as used in zope.testing's unit tests) was very painful. To quote jwz: Some people, when confronted with a problem, think "I know, I’ll use regular expressions." Now they have two problems Normalizing output might be the easier way when you initially write the test. But when some time later your test starts failing, they're a nightmare to debug. Marius Gedminas -- Any sufficiently advanced technology is indistinguishable from a rigged demo. - Andy Finkel, computer guy
On Sun, Mar 9, 2008 at 7:54 PM, Gary Poster <gary@zope.com> wrote:
I did have one somewhat trivial thought. I generally prefer durations and intervals expressed as datetime.timedeltas myself, because they convey their meaning without having to look it up docs (is that number value a number of seconds? milliseconds? minutes?). There might even be a zcml built in for schema field for that; I believe I remember that there is in ZConfig.
ZConfig supports two datatypes for time deltas: "time-interval" is a number of seconds, provided as a float. In a configuration file, a slightly more readable expression is used. For example, "4d 3h 2m 1s" means "4 days 3 hours 2 minutes 1 second". "timedelta" is a datetime.timedelta value. The syntax in the configuration file is the same as for time-interval. Neither allows specifying a negative delta value; they're really about an interval rather than a delta. Fortunately, that covers most configuration uses. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
Hi! On Sun, 2008-03-09 at 20:54 -0400, Gary Poster wrote:
I did have one somewhat trivial thought. I generally prefer durations and intervals expressed as datetime.timedeltas myself, because they convey their meaning without having to look it up docs (is that number value a number of seconds? milliseconds? minutes?). There might even be a zcml built in for schema field for that; I believe I remember that there is in ZConfig.
Minor detail - yes it has warts - there are other things that will need doing to, its just time that this code was put out there.
Also, some variety of doctest would be nice. Even when a package is not using doctests, I add new tests as doctest unless there's a really good reason not to.
It was the only way I could debug and introspect the problems from the Unit tests - the python debugger does not run properly from the doctests, and the original test are all unit tests...
In this case, it looks like you've made the code significantly more robust, which has added some probably necessary complexity. The code looks readable, but I recommend a maintainer-oriented overview/ introduction as a doctest, at the least. For instance, perhaps you could think about documentation about the rationale for the approach and about the dance that this code participates in (with the lock files and all the possible SMTP error conditions and the code's responses). Of course, even more friendly docs than that would be nice, but I'm only asking for what I myself tend to give, unfortunately.
A little does need adding there - I have scanned state machine diagrams and the design rationale. I will go and put those in the checkin when I have a spare moment. Regards, Matthew
On Mar 11, 2008, at 5:01 PM, Matthew Grant wrote:
Hi!
On Sun, 2008-03-09 at 20:54 -0400, Gary Poster wrote:
I did have one somewhat trivial thought. I generally prefer durations and intervals expressed as datetime.timedeltas myself, because they convey their meaning without having to look it up docs (is that number value a number of seconds? milliseconds? minutes?). There might even be a zcml built in for schema field for that; I believe I remember that there is in ZConfig.
Minor detail - yes it has warts - there are other things that will need doing to, its just time that this code was put out there.
Completely agree.
Also, some variety of doctest would be nice. Even when a package is not using doctests, I add new tests as doctest unless there's a really good reason not to.
It was the only way I could debug and introspect the problems from the Unit tests - the python debugger does not run properly from the doctests, and the original test are all unit tests...
Understood. For future reference, if you out the set_trace on the same line, you *can* use pdb. For instance, this would let you walk into the ``do_foo`` function. >>> import pdb; pdb.set_trace(); do_foo()
In this case, it looks like you've made the code significantly more robust, which has added some probably necessary complexity. The code looks readable, but I recommend a maintainer-oriented overview/ introduction as a doctest, at the least. For instance, perhaps you could think about documentation about the rationale for the approach and about the dance that this code participates in (with the lock files and all the possible SMTP error conditions and the code's responses). Of course, even more friendly docs than that would be nice, but I'm only asking for what I myself tend to give, unfortunately.
A little does need adding there - I have scanned state machine diagrams and the design rationale. I will go and put those in the checkin when I have a spare moment.
Sounds like a great start. Thanks for your work. Gary
On Sun, Mar 09, 2008 at 08:54:11PM -0400, Gary Poster wrote:
I did a five-minute skim of the checkin but hope to look a bit more tomorrow. Hopefully Marius, Benji, Albertas, or someone else who has actually done work on this package will take a look and chime in.
I intend to. Actually, I feel the obligation to. Sadly, the feeling of obligation usually leads me to procrastinate and do something completely different... Marius Gedminas -- The BeOS takes the best features from the major operating systems. It's got the power and flexibility of Unix, the interface and ease of use of the MacOS, and Minesweeper from Windows. -- Tyler Riti
participants (16)
-
Andreas Jung -
Benji York -
David Pratt -
Dieter Maurer -
Fred Drake -
Gary Poster -
Jens Vagelpohl -
Jim Fulton -
Malthe Borch -
Marius Gedminas -
Martijn Pieters -
Matthew Grant -
Paul Winkler -
Stephan Richter -
Tres Seaver -
Wichert Akkerman