From holger at merlinux.eu Fri Mar 1 10:50:19 2013 From: holger at merlinux.eu (holger krekel) Date: Fri, 1 Mar 2013 09:50:19 +0000 Subject: [pytest-dev] tox-1.4.3: quickstart command, "-l" option and fixes Message-ID: <20130301095019.GF9677@merlinux.eu> tox 1.4.3: the Python virtualenv-based testing automatizer ============================================================================= tox 1.4.3 fixes some bugs and introduces a new script and two new options: - "tox-quickstart" - run this script, answer a few questions, and get a tox.ini created for you (thanks Marc Abramowitz) - "tox -l" lists configured environment names (thanks Lukasz Balcerzak) - (experimental) "--installpkg=localpath" option which will skip the sdist-creation of a package and instead install the given localpath package. - use pip-script.py instead of pip.exe on win32 to avoid windows locking the .exe Note that the sister project "detox" should continue to work - it's a separately released project which drives tox test runs on multiple CPUs in parallel. More documentation: http://tox.testrun.org/ Installation: pip install -U tox repository hosting and issue tracking on bitbucket: https://bitbucket.org/hpk42/tox What is tox? ---------------- tox standardizes and automates tedious python driven test activities driven from a simple ``tox.ini`` file, including: * creation and management of different virtualenv environments with different Python interpreters * packaging and installing your package into each of them * running your test tool of choice, be it nose, py.test or unittest2 or other tools such as "sphinx" doc checks * testing dev packages against each other without needing to upload to PyPI best, Holger Krekel CHANGELOG ================ 1.4.3 (compared to 1.4.2) -------------------------------- - introduce -l|--listenv option to list configured environments (thanks Lukasz Balcerzak) - fix downloadcache determination to work according to docs: Only make pip use a download cache if PIP_DOWNLOAD_CACHE or a downloadcache=PATH testenv setting is present. (The ENV setting takes precedence) - fix issue84 - pypy on windows creates a bin not a scripts venv directory (thanks Lukasz Balcerzak) - experimentally introduce --installpkg=PATH option to install a package rather than create/install an sdist package. This will still require and use tox.ini and tests from the current working dir (and not from the remote package). - substitute {envsitepackagesdir} with the package installation directory (closes #72) (thanks g2p) - issue #70 remove PYTHONDONTWRITEBYTECODE workaround now that virtualenv behaves properly (thanks g2p) - merged tox-quickstart command, contributed by Marc Abramowitz, which generates a default tox.ini after asking a few questions - fix #48 - win32 detection of pypy and other interpreters that are on PATH (thanks Gustavo Picon) - fix grouping of index servers, it is now done by name instead of indexserver url, allowing to use it to separate dependencies into groups even if using the same default indexserver. - look for "tox.ini" files in parent dirs of current dir (closes #34) - the "py" environment now by default uses the current interpreter (sys.executable) make tox' own setup.py test execute tests with it (closes #46) - change tests to not rely on os.path.expanduser (closes #60), also make mock session return args[1:] for more precise checking (closes #61) thanks to Barry Warszaw for both. From alfredodeza at gmail.com Wed Mar 6 16:04:18 2013 From: alfredodeza at gmail.com (Alfredo Deza) Date: Wed, 6 Mar 2013 10:04:18 -0500 Subject: [pytest-dev] py.test foreground service listener Message-ID: I was looking into passing a path (which can contain some options as well) to a foreground py.test process that is "listening" for commands to run. This would help integrate text editors easily send a non-blocking test to the terminal so that further interaction can take place there. Complex tests that take very long (or need debugging via pdb) can be very problematic to run from an editor. Most editors don't have a terminal emulator, so it is either cumbersome or impossible to jump from the editor to the shell. Having a foreground py.test process would help here. The context would look similar to this: $ py.test --foreground --listen=/some/socket ============= foreground process starts ============= platform darwin -- Python 2.7.2 -- pytest-2.3.4 listening on: /some/socket And then from an editor some utility would send the desired path with options: utility '-s /path/to/test_file.py' --socket=/some/socket The listener would then receive the above command and execute: received test task collected 1 item /path/to/test_file.py . ========== 1 passed in 0.01 seconds ========== And then it would again keep listening for more "tasks". The scenario above is very simple, but as things get complicated, with more options (like dropping to pdb) it gets more useful. On IRC Ronny mentioned that this would be very tricky to accomplish (probably issues with the import cache). So this would mean that currently, a plugin would just not be suitable? Are there any options that would be left for such a thing? If not coming directly out of py.test maybe a wrapper would probably work here but I am not aware of any tool that may help. Any ideas/feedback would be great. -Alfredo -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ronny.Pfannschmidt at gmx.de Wed Mar 6 16:45:41 2013 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Wed, 06 Mar 2013 16:45:41 +0100 Subject: [pytest-dev] py.test foreground service listener In-Reply-To: References: Message-ID: <513764A5.1070903@gmx.de> Hello Alfredo, on closer inspection the following scheme may fit, it is semilar to the current looponfail mode but more simple to handle (i.e. without execnet) instead of running an actual pytest session you can hook into pytest_main, and have a process that just reads lines of json from the socket, then passes those as args to a subprocess call to py.test this also seems like an interesting mode of operation for tox (for posargs) -- Ronny On 03/06/2013 04:04 PM, Alfredo Deza wrote: > I was looking into passing a path (which can contain some options as > well) to a foreground py.test process that is "listening" for commands > to run. > > This would help integrate text editors easily send a non-blocking test > to the terminal so that further interaction can take place there. > Complex tests that take very long (or need debugging via pdb) can be > very problematic to run from an editor. > > Most editors don't have a terminal emulator, so it is either cumbersome > or impossible to jump from the editor to the shell. Having a foreground > py.test process would help here. > > The context would look similar to this: > > $ py.test --foreground --listen=/some/socket > ============= foreground process starts ============= > platform darwin -- Python 2.7.2 -- pytest-2.3.4 > listening on: /some/socket > > And then from an editor some utility would send the desired path with > options: > > utility '-s /path/to/test_file.py' --socket=/some/socket > > The listener would then receive the above command and execute: > > received test task > collected 1 item > > /path/to/test_file.py . > ========== 1 passed in 0.01 seconds ========== > > And then it would again keep listening for more "tasks". The scenario > above is very simple, but as things get complicated, with more options > (like dropping to pdb) it gets more useful. > > On IRC Ronny mentioned that this would be very tricky to accomplish > (probably issues with the import cache). So this would mean that > currently, a plugin would just not be suitable? > > Are there any options that would be left for such a thing? If not coming > directly out of py.test maybe a wrapper would probably work here but I > am not aware of any tool that may help. > > Any ideas/feedback would be great. > > > -Alfredo > > > > _______________________________________________ > Pytest-dev mailing list > Pytest-dev at python.org > http://mail.python.org/mailman/listinfo/pytest-dev From holger at merlinux.eu Wed Mar 6 19:40:05 2013 From: holger at merlinux.eu (holger krekel) Date: Wed, 6 Mar 2013 18:40:05 +0000 Subject: [pytest-dev] py.test foreground service listener In-Reply-To: References: Message-ID: <20130306184005.GI9677@merlinux.eu> On Wed, Mar 06, 2013 at 10:04 -0500, Alfredo Deza wrote: > I was looking into passing a path (which can contain some options as well) > to a foreground py.test process that is "listening" for commands to run. > > This would help integrate text editors easily send a non-blocking test to > the terminal so that further interaction can take place there. Complex > tests that take very long (or need debugging via pdb) can be very > problematic to run from an editor. > > Most editors don't have a terminal emulator, so it is either cumbersome or > impossible to jump from the editor to the shell. Having a foreground > py.test process would help here. > > The context would look similar to this: > > $ py.test --foreground --listen=/some/socket > ============= foreground process starts ============= > platform darwin -- Python 2.7.2 -- pytest-2.3.4 > listening on: /some/socket > > And then from an editor some utility would send the desired path with > options: > > utility '-s /path/to/test_file.py' --socket=/some/socket > > The listener would then receive the above command and execute: > > received test task > collected 1 item > > /path/to/test_file.py . > ========== 1 passed in 0.01 seconds ========== > > And then it would again keep listening for more "tasks". The scenario above > is very simple, but as things get complicated, with more options (like > dropping to pdb) it gets more useful. > > On IRC Ronny mentioned that this would be very tricky to accomplish > (probably issues with the import cache). So this would mean that currently, > a plugin would just not be suitable? As ronny already hints at, i think the listening foreground process would receive test paths to run along with options, but it would not even import "pytest" or any of the test files itself. Instead it invokes a "py.test [options] testpaths" subprocess (passing through stdin/stdout/stderr) and passing back return values. Such a listener process could be rather simple and probably shouldn't be a plugin (after all we don't even want to "import pytest" in it). It should rather be a "pytest-wrapserver" script and i assume it would be a rather simple thing to do, no changes to pytest required. Unless i am missing something, of course :) holger > > Are there any options that would be left for such a thing? If not coming > directly out of py.test maybe a wrapper would probably work here but I am > not aware of any tool that may help. > > Any ideas/feedback would be great. > > > -Alfredo > _______________________________________________ > Pytest-dev mailing list > Pytest-dev at python.org > http://mail.python.org/mailman/listinfo/pytest-dev From Ronny.Pfannschmidt at gmx.de Wed Mar 6 20:56:55 2013 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Wed, 06 Mar 2013 20:56:55 +0100 Subject: [pytest-dev] py.test foreground service listener In-Reply-To: <20130306184005.GI9677@merlinux.eu> References: <20130306184005.GI9677@merlinux.eu> Message-ID: <51379F87.40508@gmx.de> Hi Holger, Alfredo, if its going to be that generic, then a fit name might be cmdrun-d as a generic utility that listens for shell escaped lines on a socket/stdin and executes them with stdout/err of the shell if that mode of operation is used, the tool could be as simple as a #!/usr/bin/python import sys, shlex, subprocess for line in sys.stdin: cmd = shlex.split(line) result = subprocess.call(cmd) if result: print(str(cmd) + "Failed") # -- ronny On 03/06/2013 07:40 PM, holger krekel wrote: > On Wed, Mar 06, 2013 at 10:04 -0500, Alfredo Deza wrote: >> I was looking into passing a path (which can contain some options as well) >> to a foreground py.test process that is "listening" for commands to run. >> >> This would help integrate text editors easily send a non-blocking test to >> the terminal so that further interaction can take place there. Complex >> tests that take very long (or need debugging via pdb) can be very >> problematic to run from an editor. >> >> Most editors don't have a terminal emulator, so it is either cumbersome or >> impossible to jump from the editor to the shell. Having a foreground >> py.test process would help here. >> >> The context would look similar to this: >> >> $ py.test --foreground --listen=/some/socket >> ============= foreground process starts ============= >> platform darwin -- Python 2.7.2 -- pytest-2.3.4 >> listening on: /some/socket >> >> And then from an editor some utility would send the desired path with >> options: >> >> utility '-s /path/to/test_file.py' --socket=/some/socket >> >> The listener would then receive the above command and execute: >> >> received test task >> collected 1 item >> >> /path/to/test_file.py . >> ========== 1 passed in 0.01 seconds ========== >> >> And then it would again keep listening for more "tasks". The scenario above >> is very simple, but as things get complicated, with more options (like >> dropping to pdb) it gets more useful. >> >> On IRC Ronny mentioned that this would be very tricky to accomplish >> (probably issues with the import cache). So this would mean that currently, >> a plugin would just not be suitable? > > As ronny already hints at, i think the listening foreground process > would receive test paths to run along with options, but it would not > even import "pytest" or any of the test files itself. Instead it invokes > a "py.test [options] testpaths" subprocess (passing through > stdin/stdout/stderr) and passing back return values. > > Such a listener process could be rather simple and probably shouldn't > be a plugin (after all we don't even want to "import pytest" in it). > It should rather be a "pytest-wrapserver" script and i assume it would > be a rather simple thing to do, no changes to pytest required. > > Unless i am missing something, of course :) > > holger > > >> >> Are there any options that would be left for such a thing? If not coming >> directly out of py.test maybe a wrapper would probably work here but I am >> not aware of any tool that may help. >> >> Any ideas/feedback would be great. >> >> >> -Alfredo > >> _______________________________________________ >> Pytest-dev mailing list >> Pytest-dev at python.org >> http://mail.python.org/mailman/listinfo/pytest-dev > > _______________________________________________ > Pytest-dev mailing list > Pytest-dev at python.org > http://mail.python.org/mailman/listinfo/pytest-dev From brianna.laugher at gmail.com Tue Mar 19 07:18:39 2013 From: brianna.laugher at gmail.com (Brianna Laugher) Date: Tue, 19 Mar 2013 17:18:39 +1100 Subject: [pytest-dev] expand comparison diff? Message-ID: Hi, Despite being touted as one of its strengths, I think the output on comparison of complex objects could use some work. I work with many tests that compare dictionaries for example ('assert result == expected'), and if they fail I usually have to dig around, manually iterate through them etc to figure out where the difference is. Here is an example. tests/unit/formatters/test_WxPhrase.py:638: in test_mergeBackSeparableAttributesMultipleSentencesOK > assert result == expected E assert {frozenset(['...00:00 GMT))]]} == {frozenset(['T...00:00 GMT))]]} E Skipping 161 identical leading characters in diff E 00 GMT))]]} Note using --tb=long does not expand the diff text -- it just expands the traceback stack which is not helpful in this case. If I use the compare() method in testfixtures library instead of a plain assert statement, the output is a bit better: tests/unit/formatters/test_WxPhrase.py:639: in test_mergeBackSeparableAttributesMultipleSentencesOK > compare(result, expected) ../GFESuite-Builds/Release/release/lib/python2.7/site-packages/testfixtures/comparison.py:268: in compare > raise AssertionError(message) E AssertionError: dict not as expected: E E values differ: E frozenset(['TS']): [[WxStatBlock((Jan 01 70 16:00:00 GMT, Jan 01 70 22:00:00 GMT))], E [WxStatBlock((Jan 01 70 10:00:00 GMT, Jan 01 70 16:00:00 GMT))]] != [[WxStatBlock((Jan 01 70 16:00:00 GMT, Jan 01 70 22:00:00 GMT))], E [WxStatBlock((Jan 01 70 10:00:00 GMT, Jan 01 70 16:00:00 GMT))]] OK - now I can see that the assert is failing because the order is different. Is there a way, or could one be added, to be more verbose in object (esp. list/set/dict) comparison? An option like --cmp=verbose (or maybe it makes sense incorporated into --tb=long??) I had a half-hearted attempt at writing my own pytest_assertrepr_compare to use this compare() method (I really like using natural assert statements) but py.test didn't seem to pick it up. I might investigate this further but also maybe this is a good area for pytest to improve? thanks, Brianna -- They've just been waiting in a mountain for the right moment: http://modernthings.org/ From brianna.laugher at gmail.com Tue Mar 19 07:29:47 2013 From: brianna.laugher at gmail.com (Brianna Laugher) Date: Tue, 19 Mar 2013 17:29:47 +1100 Subject: [pytest-dev] expand comparison diff? In-Reply-To: References: Message-ID: On 19 March 2013 17:18, Brianna Laugher wrote: > tests/unit/formatters/test_WxPhrase.py:639: in > test_mergeBackSeparableAttributesMultipleSentencesOK >> compare(result, expected) > ../GFESuite-Builds/Release/release/lib/python2.7/site-packages/testfixtures/comparison.py:268: > in compare >> raise AssertionError(message) > E AssertionError: dict not as expected: > E > E values differ: > E frozenset(['TS']): [[WxStatBlock((Jan 01 70 16:00:00 GMT, Jan > 01 70 22:00:00 GMT))], > E [WxStatBlock((Jan 01 70 10:00:00 GMT, Jan 01 70 16:00:00 > GMT))]] != [[WxStatBlock((Jan 01 70 16:00:00 GMT, Jan 01 70 22:00:00 > GMT))], > E [WxStatBlock((Jan 01 70 10:00:00 GMT, Jan 01 70 16:00:00 GMT))]] > > OK - now I can see that the assert is failing because the order is different. Er, this is obviously not actually the case - serves me right for having incomplete __repr__ methods. But I still maintain it is more useful output than what pytest currently gives. Brianna -- They've just been waiting in a mountain for the right moment: http://modernthings.org/ From Ronny.Pfannschmidt at gmx.de Tue Mar 19 08:57:09 2013 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Tue, 19 Mar 2013 08:57:09 +0100 Subject: [pytest-dev] expand comparison diff? In-Reply-To: References: Message-ID: <51481A55.80206@gmx.de> Hi Brianna, i see what you mean, can you file an issue on bitbucket -- Ronny On 03/19/2013 07:18 AM, Brianna Laugher wrote: > Hi, > > Despite being touted as one of its strengths, I think the output on > comparison of complex objects could use some work. I work with many > tests that compare dictionaries for example ('assert result == > expected'), and if they fail I usually have to dig around, manually > iterate through them etc to figure out where the difference is. > > Here is an example. > > tests/unit/formatters/test_WxPhrase.py:638: in > test_mergeBackSeparableAttributesMultipleSentencesOK >> assert result == expected > E assert {frozenset(['...00:00 GMT))]]} == {frozenset(['T...00:00 GMT))]]} > E Skipping 161 identical leading characters in diff > E 00 GMT))]]} > > Note using --tb=long does not expand the diff text -- it just expands > the traceback stack which is not helpful in this case. > > If I use the compare() method in testfixtures library instead of a > plain assert statement, the output is a bit better: > > tests/unit/formatters/test_WxPhrase.py:639: in > test_mergeBackSeparableAttributesMultipleSentencesOK >> compare(result, expected) > ../GFESuite-Builds/Release/release/lib/python2.7/site-packages/testfixtures/comparison.py:268: > in compare >> raise AssertionError(message) > E AssertionError: dict not as expected: > E > E values differ: > E frozenset(['TS']): [[WxStatBlock((Jan 01 70 16:00:00 GMT, Jan > 01 70 22:00:00 GMT))], > E [WxStatBlock((Jan 01 70 10:00:00 GMT, Jan 01 70 16:00:00 > GMT))]] != [[WxStatBlock((Jan 01 70 16:00:00 GMT, Jan 01 70 22:00:00 > GMT))], > E [WxStatBlock((Jan 01 70 10:00:00 GMT, Jan 01 70 16:00:00 GMT))]] > > OK - now I can see that the assert is failing because the order is different. > > Is there a way, or could one be added, to be more verbose in object > (esp. list/set/dict) comparison? An option like --cmp=verbose (or > maybe it makes sense incorporated into --tb=long??) > > I had a half-hearted attempt at writing my own > pytest_assertrepr_compare to use this compare() method (I really like > using natural assert statements) but py.test didn't seem to pick it > up. I might investigate this further but also maybe this is a good > area for pytest to improve? > > thanks, > Brianna > From brianna.laugher at gmail.com Wed Mar 20 01:41:58 2013 From: brianna.laugher at gmail.com (Brianna Laugher) Date: Wed, 20 Mar 2013 11:41:58 +1100 Subject: [pytest-dev] expand comparison diff? In-Reply-To: <51481A55.80206@gmx.de> References: <51481A55.80206@gmx.de> Message-ID: Sure - https://bitbucket.org/hpk42/pytest/issue/279/output-on-assertion-failure-of-object On 19 March 2013 18:57, Ronny Pfannschmidt wrote: > Hi Brianna, > > i see what you mean, > can you file an issue on bitbucket > > -- Ronny > > > On 03/19/2013 07:18 AM, Brianna Laugher wrote: >> >> Hi, >> >> Despite being touted as one of its strengths, I think the output on >> comparison of complex objects could use some work. I work with many >> tests that compare dictionaries for example ('assert result == >> expected'), and if they fail I usually have to dig around, manually >> iterate through them etc to figure out where the difference is. >> >> Here is an example. >> >> tests/unit/formatters/test_WxPhrase.py:638: in >> test_mergeBackSeparableAttributesMultipleSentencesOK >>> >>> assert result == expected >> >> E assert {frozenset(['...00:00 GMT))]]} == {frozenset(['T...00:00 >> GMT))]]} >> E Skipping 161 identical leading characters in diff >> E 00 GMT))]]} >> >> Note using --tb=long does not expand the diff text -- it just expands >> the traceback stack which is not helpful in this case. >> >> If I use the compare() method in testfixtures library instead of a >> plain assert statement, the output is a bit better: >> >> tests/unit/formatters/test_WxPhrase.py:639: in >> test_mergeBackSeparableAttributesMultipleSentencesOK >>> >>> compare(result, expected) >> >> >> ../GFESuite-Builds/Release/release/lib/python2.7/site-packages/testfixtures/comparison.py:268: >> in compare >>> >>> raise AssertionError(message) >> >> E AssertionError: dict not as expected: >> E >> E values differ: >> E frozenset(['TS']): [[WxStatBlock((Jan 01 70 16:00:00 GMT, Jan >> 01 70 22:00:00 GMT))], >> E [WxStatBlock((Jan 01 70 10:00:00 GMT, Jan 01 70 16:00:00 >> GMT))]] != [[WxStatBlock((Jan 01 70 16:00:00 GMT, Jan 01 70 22:00:00 >> GMT))], >> E [WxStatBlock((Jan 01 70 10:00:00 GMT, Jan 01 70 16:00:00 GMT))]] >> >> OK - now I can see that the assert is failing because the order is >> different. >> >> Is there a way, or could one be added, to be more verbose in object >> (esp. list/set/dict) comparison? An option like --cmp=verbose (or >> maybe it makes sense incorporated into --tb=long??) >> >> I had a half-hearted attempt at writing my own >> pytest_assertrepr_compare to use this compare() method (I really like >> using natural assert statements) but py.test didn't seem to pick it >> up. I might investigate this further but also maybe this is a good >> area for pytest to improve? >> >> thanks, >> Brianna >> > -- They've just been waiting in a mountain for the right moment: http://modernthings.org/