unittest isolation and warnings
Hi guys, when writing tests, I suddenly discovered that unittest is not isolated to warnings. Example: One of my tests emits warnings when a certain condition is met. Instead of reporting the error immediately, it uses warnings, and at the end of the test, an error is produced if there were warnings. if hasattr(__main__, "__warningregistry__"): raise RuntimeError("There are errors, see above.") By chance, I discovered that an error was suddenly triggered without a warning. That must mean the warning existed already from another test as a left-over. My question: Is that known, and is that intended? To what extent are the test cases isolated from each other? I do admit that my usage of warnings is somewhat special. But it is very convenient to report many errors on remote servers. Cheers -- Chris -- Christian Tismer :^) tismer@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : https://github.com/PySide 14482 Potsdam : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023
Tests are not isolated from the warnings system, so things will leak out.
Your best option is to use the context manager in the warnings module to
temporarily make all warnings raise exceptions and test for the exception
(I'm at the airport, hence why I don't know the name of the context
manager; the warnings module docs actually have a sample on how best to
write tests the involve warnings).
On Fri, Nov 17, 2017, 01:34 Christian Tismer,
Hi guys,
when writing tests, I suddenly discovered that unittest is not isolated to warnings.
Example: One of my tests emits warnings when a certain condition is met. Instead of reporting the error immediately, it uses warnings, and at the end of the test, an error is produced if there were warnings.
if hasattr(__main__, "__warningregistry__"): raise RuntimeError("There are errors, see above.")
By chance, I discovered that an error was suddenly triggered without a warning. That must mean the warning existed already from another test as a left-over.
My question: Is that known, and is that intended? To what extent are the test cases isolated from each other?
I do admit that my usage of warnings is somewhat special. But it is very convenient to report many errors on remote servers.
Cheers -- Chris
-- Christian Tismer :^) tismer@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : https://github.com/PySide 14482 Potsdam : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
Thanks a lot! Good to know. Ciao -- Chris On 17.11.17 16:44, Brett Cannon wrote:
Tests are not isolated from the warnings system, so things will leak out. Your best option is to use the context manager in the warnings module to temporarily make all warnings raise exceptions and test for the exception (I'm at the airport, hence why I don't know the name of the context manager; the warnings module docs actually have a sample on how best to write tests the involve warnings).
On Fri, Nov 17, 2017, 01:34 Christian Tismer,
mailto:tismer@stackless.com> wrote: Hi guys,
when writing tests, I suddenly discovered that unittest is not isolated to warnings.
Example: One of my tests emits warnings when a certain condition is met. Instead of reporting the error immediately, it uses warnings, and at the end of the test, an error is produced if there were warnings.
if hasattr(__main__, "__warningregistry__"): raise RuntimeError("There are errors, see above.")
By chance, I discovered that an error was suddenly triggered without a warning. That must mean the warning existed already from another test as a left-over.
My question: Is that known, and is that intended? To what extent are the test cases isolated from each other?
I do admit that my usage of warnings is somewhat special. But it is very convenient to report many errors on remote servers.
Cheers -- Chris
-- Christian Tismer :^) tismer@stackless.com mailto:tismer@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : https://github.com/PySide 14482 Potsdam : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023
_______________________________________________ Python-Dev mailing list Python-Dev@python.org mailto:Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
-- Christian Tismer :^) tismer@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : https://github.com/PySide 14482 Potsdam : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023
Hi Christian,
On Fri, 17 Nov 2017 10:15:24 +0100
Christian Tismer
Example: One of my tests emits warnings when a certain condition is met. Instead of reporting the error immediately, it uses warnings, and at the end of the test, an error is produced if there were warnings.
I suggest you try using subtests. This would allow you to report several errors from a given test method. https://docs.python.org/3/library/unittest.html#distinguishing-test-iteratio... Regards Antoine.
participants (3)
-
Antoine Pitrou
-
Brett Cannon
-
Christian Tismer