Does trace modules have a unit test?
In reviewing http://bugs.python.org/issue9282 the issue came up, where is the unit test for trace.py? test/test_trace.py is actually a test of the line trace facility of sys.settrace (and should have been called test_linetrace or test_settrace). The only trace import Eli could find in Lib/test is the one in regrtest.py for the -T option. Have we missed something? If you say 'make one', what should be its name? Can the existing file be renamed (I gather this can be a bit of a problem)? -- Terry Jan Reedy
On Mon, Jul 19, 2010 at 07:02, Terry Reedy <tjreedy@udel.edu> wrote:
In reviewing http://bugs.python.org/issue9282 the issue came up, where is the unit test for trace.py?
test/test_trace.py is actually a test of the line trace facility of sys.settrace (and should have been called test_linetrace or test_settrace). The only trace import Eli could find in Lib/test is the one in regrtest.py for the -T option. Have we missed something?
If you say 'make one', what should be its name? Can the existing file be renamed (I gather this can be a bit of a problem)?
I'd like to ask pydev another question on this issue. When thinking about adding unit tests for trace.py, I quickly realized it's not a usual module. Most of its output is either tracing reports to stdout, or even worse, to special files named <modname>.cover stdout output can be captured, but what about the .cover files? Can a Python unit test create temporary files in tmp/ (or somewhere else) as part of its testing, or is this forbidden? Eli
On Mon, Jul 19, 2010 at 12:12 AM, Eli Bendersky <eliben@gmail.com> wrote: ..
stdout output can be captured, but what about the .cover files? Can a Python unit test create temporary files in tmp/ (or somewhere else) as part of its testing, or is this forbidden?
That's perfectly fine. Grep in the Lib/test directory for 'tempfile.mkdtemp()' to see examples.
On Mon, Jul 19, 2010 at 12:21 AM, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
On Mon, Jul 19, 2010 at 12:12 AM, Eli Bendersky <eliben@gmail.com> wrote: ..
stdout output can be captured, but what about the .cover files? Can a Python unit test create temporary files in tmp/ (or somewhere else) as part of its testing, or is this forbidden?
That's perfectly fine. Grep in the Lib/test directory for 'tempfile.mkdtemp()' to see examples.
Actually the first hit is Lib/test/script_helper.py which contains several utilities that you may find useful.
On Mon, Jul 19, 2010 at 2:23 PM, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
On Mon, Jul 19, 2010 at 12:21 AM, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
On Mon, Jul 19, 2010 at 12:12 AM, Eli Bendersky <eliben@gmail.com> wrote: ..
stdout output can be captured, but what about the .cover files? Can a Python unit test create temporary files in tmp/ (or somewhere else) as part of its testing, or is this forbidden?
That's perfectly fine. Grep in the Lib/test directory for 'tempfile.mkdtemp()' to see examples.
Actually the first hit is Lib/test/script_helper.py which contains several utilities that you may find useful.
Yeah, the script_helper.temp_dir CM is especially handy for mucking about on the filesystem. test_cmd_line_script.py and test_runpy.py are other places to look for inspiration on ways to make heavy use of the filesystem for testing without leaving junk behind when the test is over (script_helper was actually born by refactoring the helpers from those two test suites out to a common module). I really should get back to the issue about a temp_dir equivalent worthy of inclusion in the tempfile module at some point, though (there's an unfortunate corner case relating to interpreter shutdown and generators or explicit invocation of __enter__ that causes problems for the script_helper.temp_dir incarnation. The test suite avoids that corner case naturally, but a tempfile.tempdir CM needs to cope with it) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I've opened issue 9315 (http://bugs.python.org/issue9315) to address the lack of unit tests for the trace.py module. I hope to get to it in a few days and add some tests. As Terry wrote in the beginning of this thread, Lib/test/test_trace.py currently tests the sys.settrace module, so the tests of trace.py should find a new home. Does Lib/test/test_trace_module.py make sense or is something else preferable? Eli
On Tue, Jul 20, 2010 at 10:51 AM, Eli Bendersky <eliben@gmail.com> wrote:
As Terry wrote in the beginning of this thread, Lib/test/test_trace.py currently tests the sys.settrace module, so the tests of trace.py should find a new home. Does Lib/test/test_trace_module.py make sense or is something else preferable?
IMO you should just rename test_trace.py to test_settrace.py, and put the trace.py tests in test_trace.py. Reid
On Wed, Jul 21, 2010 at 4:18 AM, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
On Tue, Jul 20, 2010 at 2:12 PM, Reid Kleckner <reid.kleckner@gmail.com> wrote: ..
IMO you should just rename test_trace.py to test_settrace.py, and put the trace.py tests in test_trace.py.
+1
Yep, we're pretty free to rename stuff as required inside the test suite. The only real exception is test.support, as that's a documented standard library module. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Jul 20, 2010 at 6:19 PM, Nick Coghlan <ncoghlan@gmail.com> wrote: ..
Yep, we're pretty free to rename stuff as required inside the test suite. The only real exception is test.support, as that's a documented standard library module.
Would anyone object to moving Lib/test/test_trace.py to Lib/test/test_line_tracing.py in 2.7 maint branch as Benjamin did in py3k? As I suggested in issue 9315 [1], the tests should be first written for 2.7 and either designed to run unchanged in 3.x, or ported in a controlled fashion to assure that we don't have regressions. (And in fact we do! [2, 3]) PS: The bikeshedding period is probably closed on this, but I would rather call the new test file test_sys_settrace rather than test_line_tracing. The sys.settrace function does more than line tracing and the test naming convention seems to be component rather than functionality oriented in the rest of the test suite. [1] http://bugs.python.org/issue9315 [2] http://bugs.python.org/issue9317 [3] http://bugs.python.org/issue9323
2010/7/21 Alexander Belopolsky <alexander.belopolsky@gmail.com>:
On Tue, Jul 20, 2010 at 6:19 PM, Nick Coghlan <ncoghlan@gmail.com> wrote: ..
Yep, we're pretty free to rename stuff as required inside the test suite. The only real exception is test.support, as that's a documented standard library module.
Would anyone object to moving Lib/test/test_trace.py to Lib/test/test_line_tracing.py in 2.7 maint branch as Benjamin did in py3k? As I suggested in issue 9315 [1], the tests should be first written for 2.7 and either designed to run unchanged in 3.x, or ported in a controlled fashion to assure that we don't have regressions. (And in fact we do! [2, 3])
PS: The bikeshedding period is probably closed on this, but I would rather call the new test file test_sys_settrace rather than test_line_tracing. The sys.settrace function does more than line tracing and the test naming convention seems to be component rather than functionality oriented in the rest of the test suite.
True, but the tests in that file are (mostly?) all about line tracing. -- Regards, Benjamin
On 7/21/2010 6:45 PM, Alexander Belopolsky wrote:
On Wed, Jul 21, 2010 at 1:42 PM, Benjamin Peterson <benjamin@python.org> wrote: ..
True, but the tests in that file are (mostly?) all about line tracing.
Hopefully this will change soon. :-)
At which point the renaming would be justified. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/
On Wed, Jul 21, 2010 at 9:26 PM, Steve Holden <steve@holdenweb.com> wrote:
On 7/21/2010 6:45 PM, Alexander Belopolsky wrote:
On Wed, Jul 21, 2010 at 1:42 PM, Benjamin Peterson <benjamin@python.org> wrote: ..
True, but the tests in that file are (mostly?) all about line tracing.
Hopefully this will change soon. :-)
At which point the renaming would be justified.
True, but at that point it may be in three or more branches while right now it is in one. It turns out, however, that the tests in the file formerly known as test_trace cover more than line tracing, for example: # Test the handling of exceptions raised by each kind of trace event. def test_call(self): self.run_test_for_event('call') def test_line(self): self.run_test_for_event('line') def test_return(self): self.run_test_for_event('return') def test_exception(self): self.run_test_for_event('exception') Furthermore, the now out of date comment in test_sys says: # testing sys.settrace() is done in test_trace.py which makes it clear that the original intent of test_trace was to test everything settrace does, not just line tracing. In any case, it looks like my original question got shadowed by the choice of name discussion. Is it ok to rename test_trace to test_whatever in the maintenance branches? The reason is that the trace module is severely broken in 3.x and there are no regression tests in 2.7 to test the 3.x port against. Another alternative is to simply add trace module tests into test_trace. After all, the test_sys comment I mentioned is followed by # testing sys.setprofile() is done in test_profile.py The profile situation is different, though: test_profile tests the profile module, but not sys.setprofile() . Since there is some movement in this area, I don't think it would hurt to make a decision now that will help writing tests in the future. I see three solutions: 1. Minimal: do not rename test_trace in 2.7 and add trace module tests to the existing file. Whether to revert test_trace to test_line_tracing renaming in 3.2 can be decided after we have 2.7 tests written. 2. Rename test_trace in 2.7/3.1 and test_line_tracing in 3.2 to test_sys_settrace. 3. Rename test_trace in 2.7/3.1 to test_line_tracing. My preference in #2, but I can live with #1 (assuming Eli is the one who will write the tests:-).
On Fri, Jul 23, 2010 at 3:30 AM, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
I see three solutions:
1. Minimal: do not rename test_trace in 2.7 and add trace module tests to the existing file. Whether to revert test_trace to test_line_tracing renaming in 3.2 can be decided after we have 2.7 tests written. 2. Rename test_trace in 2.7/3.1 and test_line_tracing in 3.2 to test_sys_settrace. 3. Rename test_trace in 2.7/3.1 to test_line_tracing.
My preference in #2, but I can live with #1 (assuming Eli is the one who will write the tests:-).
I'd say just fix it and go with #2 (just update the comment in test_sys as well!) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Jul 22, 2010 at 6:33 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Fri, Jul 23, 2010 at 3:30 AM, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
I see three solutions:
1. Minimal: do not rename test_trace in 2.7 and add trace module tests to the existing file. Whether to revert test_trace to test_line_tracing renaming in 3.2 can be decided after we have 2.7 tests written. 2. Rename test_trace in 2.7/3.1 and test_line_tracing in 3.2 to test_sys_settrace. 3. Rename test_trace in 2.7/3.1 to test_line_tracing.
My preference in #2, but I can live with #1 (assuming Eli is the one who will write the tests:-).
I'd say just fix it and go with #2 (just update the comment in test_sys as well!)
I have looked further into setprofile tests for inspiration and discovered that it is covered in test_profilehooks. Note that sys.settrace and sys.setprofile are very similar. They take a function with the same signature and report events in the same format. The only difference is in the kind of events that get reported. I made an experiment replacing [sg]etprofile with [sg]ettrace in test-profilehooks and all tests worked reporting failures in comparing outputs. I would like to use this opportunity to merge settrace and setprofile tests. Here is the proposal: Step 1: Rename test_trace to test_sys_settrace and test_profilehooks to test_sys_setprofile. Step 2: Create tracetester helper file with abstract test cases made from the union of test_sys_settrace and test_sys_setprofile test cases and replace the concrete test cases in test_sys_set* with subclasses that define setmethod, getmethod and expected_output. An alternative proposal is to put all tests for both trace and profile into a single test_sys_tracing file. What do you think? (Another option is to leave test_profilehooks and use rename test_trace to test_tracehooks in the step 1 above.)
On Sun, Jul 25, 2010 at 2:20 AM, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
I would like to use this opportunity to merge settrace and setprofile tests. Here is the proposal:
Step 1: Rename test_trace to test_sys_settrace and test_profilehooks to test_sys_setprofile. Step 2: Create tracetester helper file with abstract test cases made from the union of test_sys_settrace and test_sys_setprofile test cases and replace the concrete test cases in test_sys_set* with subclasses that define setmethod, getmethod and expected_output.
An alternative proposal is to put all tests for both trace and profile into a single test_sys_tracing file.
What do you think?
(Another option is to leave test_profilehooks and use rename test_trace to test_tracehooks in the step 1 above.)
A tracetester helper module + the two test module renames sounds like a fine idea to me (your proposed new names should be clearer than the existing *hooks naming, given the existence of "profile" and "trace" modules). You may want to do a grep for references to the existing test files in comments in the rest of the test suite though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Jul 24, 2010 at 11:18 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Sun, Jul 25, 2010 at 2:20 AM, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote: ..
Step 1: Rename test_trace to test_sys_settrace and test_profilehooks to test_sys_setprofile. .. A tracetester helper module + the two test module renames sounds like a fine idea to me (your proposed new names should be clearer than the existing *hooks naming, given the existence of "profile" and "trace" modules).
I have implemented the step 1 above in 3.2 and 2.7 branches, but svnmerge failed in 3.1 and I suspect that this was because r79102 did not make it to release31-maint. It is my understanding that test improvements are normally backported to maintenance branches and r79102 is a test-only revision. Unfortunately, I was not able to figure out how to convince svnmerge to merge r79102 in 3.1. Can someone help?
On Mon, Jul 26, 2010 at 1:50 AM, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
On Sat, Jul 24, 2010 at 11:18 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Sun, Jul 25, 2010 at 2:20 AM, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote: ..
Step 1: Rename test_trace to test_sys_settrace and test_profilehooks to test_sys_setprofile. .. A tracetester helper module + the two test module renames sounds like a fine idea to me (your proposed new names should be clearer than the existing *hooks naming, given the existence of "profile" and "trace" modules).
I have implemented the step 1 above in 3.2 and 2.7 branches, but svnmerge failed in 3.1 and I suspect that this was because r79102 did not make it to release31-maint. It is my understanding that test improvements are normally backported to maintenance branches and r79102 is a test-only revision. Unfortunately, I was not able to figure out how to convince svnmerge to merge r79102 in 3.1. Can someone help?
Looking at r79102, it updates a number of tests to use the new assert* unittest methods that aren't available in Python 3.1. So while the patch itself is test-only, it can't be backported because it depends on new features. While the general preference is to backport tests, it is also acknowledged that that can become overly difficult as the test cases diverge. Up to you if you want to manually fix your patch for 3.1, drop the test_profilehooks changes, or else just not backport at all. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Jul 25, 2010 at 5:57 PM, Nick Coghlan <ncoghlan@gmail.com> wrote: ..
While the general preference is to backport tests, it is also acknowledged that that can become overly difficult as the test cases diverge. Up to you if you want to manually fix your patch for 3.1, drop the test_profilehooks changes, or else just not backport at all.
Thanks for explanation, Nick. Since my change is for the benefit of tests that are planned to go in 2.7, it makes little sense to skip 3.1. The change introduced by r79102 in the relevant files is trivial (s/== None/is None/). I just added it manually. See rev 83148.
participants (7)
-
Alexander Belopolsky -
Benjamin Peterson -
Eli Bendersky -
Nick Coghlan -
Reid Kleckner -
Steve Holden -
Terry Reedy