Mailman 3 October 2013 - pytest-dev

plugin status page
by holger krekel Nov. 14, 2013

Nov. 14, 2013

Hi Bruno, all, i've brought your initial work online at the "dev" pytest pages: http://pytest.org/dev/plugins_index.html I think it's a good start, thanks! My comments and wishes (others may comment as well!): - i'd collapse NAME and VERSION columns to save space, i.e. "pytest-bdd-0.6.1". - we should try to collect repository locations. maybe parsing for github/bitbucket urls would yield most of them automatically? Maybe we need to add some manually. - what about adding download numbers? - as to code organisation: you can leave it as is for now or (maybe better) put all code and generated files into a dedicated directory. A very interesting bit will be the "2.4.2 compat" and "dev" compat determination. In fact, i think "devpi" should help with that although i guess i need to improve some things. You can give using it a try with these steps: pip install devpi-client devpi use http://devpi.net/root/pypi devpi test -e py27 pytest-pep8 The last bit should download the pypi-version of pytest-pep8, unpack it and run tox for the "py27" environment and then post back test results (as a json file) to the devpi.net server. You can also ask about plugin "test" status: devpi list pytest-pep8 I just found out that this doesn't work for "pytest-flakes" yet although that does have a tox.ini. need to look into what's going on. In principle, devpi-server (an instance of which you are using above), provides a lot of detailed test info as a json, see here for example: http://devpi.net/+tests/6199353734615fde47d1fbfef1ebc737/toxresult The test results are stored as per the package MD5 at the moment. The whole test/list/json machinery of devpi needs some more work to become really nice, but it should already help as currently implemented, maybe best by using "devpi list" and filing feature requests against it :) For testing "development" versions, it should be enough to upload e.g. a "pytest-2.4.3.dev1" to a private index and then re-run the script that invokes something like the above devpi commands for all of the plugins. Now, for the plugins that don't have a tox.ini we could supply one out of band. For this "devpi test" would grow an option like "--toxini=mytox.ini" and would use that instead of expecting one in the downloaded unpacked distribution file. This tox.ini would probably just have a testenv that does "py.test --version" and "py.test --help" to see if something broke by just installing the plugin. So much for now, holger

3 23

monkeypatch.setattr of functions and from imports
by Daniel Nouri Oct. 29, 2013

Oct. 29, 2013

Dear all I have a function 'somefunc' in module 'a'. Another module 'b' imports that function with a "from import": a.py: def somefunc(): return 'hey!' b.py: from a import somefunc def someotherfunc(): return somefunc() + 'there' I want to now test 'someotherfunc' while patching away 'somefunc': def test_someotherfunc(monkeypatch): from b import someotherfunc monkeypatch.setattr('a.somefunc', lambda: 'eh?') someotherfunc() But this will fail, since 'b' imported somefunc from 'a' before we monkey patched the module's attribute. Without a "from import", this would work: b.py: import a def someotherfunc(): return a.somefunc() + 'there' What I would normally resort to is patch somefunc's func_code, so that even code that used a "from import" before I could patch will use the patched version. Thoughts? Cheers Daniel

4 6

parametrization ids simplification/generalization
by holger krekel Oct. 22, 2013

Oct. 22, 2013

Hi Brianna, all, I am thinking about constructing ids for parametrized tests using argname/valindex, i.e. not use the actual parametrization values. The latter causes some problems (among them also issue357) and generally doesn't allow to distinguish the two tests in: @pytest.mark.parametrize("arg", [1,1]) def test_some(arg): pass Currently this gives two identical test ids "test_some[1]". With the new scheme this gives: test_some[arg0] test_some[arg1] Such a scheme would guarantee unique IDs. The scheme is already used for user instances. Note that when you get a traceback you would still see the actual argument values in the traceback. Any objections? holger

2 4

Allow pytest.raises to assert about exception arguments
by Vladimir Keleshev Oct. 14, 2013

Oct. 14, 2013

Right now one of my code-bases is filled with assertions like: with raises(SpecificError) as error: some_action() assert 'specific message' == str(error) Why not change `raises` to allow instances of exceptions as well: with raises(SpecificError('specific message')): some_action() I think, this can improve readability of `raises` assertions. Can't even think of immediate downsides of this. What do you think? —Vladimir

3 2

solving the "too static" fixture scope problem
by holger krekel Oct. 13, 2013

Oct. 13, 2013

Hi pytest users and developers, I'd like to discuss and get feedback on extending the fixture mechanism and fix what i consider a biggest limitation at the moment. Problem: fixtures are tied statically to a scope ================================================= For example, you cannot use monkeypatch in a higher than "function" scoped fixture. Same is true for tmpdir and probably also many user-defined fixtures. I've certainly had this problem myself many times that i had a fixture function that didn't really care in what scope it was used. There are ways to get around this problem but they are not pretty: @pytest.fixture(scope="module") def myfix_module(request return _myfix(request) @pytest.fixture(scope="function") def myfix_function(request return _myfix(request) where _myfix is the function that doesn't care about the actual scope. Even then, you can't use builtin fixtures like "monkeypatch", "tmpdir", etc. Solution Idea: introduce "each" scoped fixtures ===================================================== The idea is allow a fixture function to declare it wants to be used in the scope of the requesting fixture function (or at function-scope if used from a test). This is how "monkeypatch" would be implemented then: @pytest.fixture(scope="each") def monkeypatch(request): ... # same implementation as in _pytest/monkeypatch.py The new "each" scope means that each fixture/test requesting the "monkeypatch" fixture would receive its own fixture instance. So a session-scoped fixture could naturally use it like this: @pytest.fixture(scope="session") def myfix(monkeypatch): monkeypatch.setattr(...) return some_value The passed in monkeypatch object here is a specific instance just for the ``myfix`` fixture function: "each" fixture function requesting ``monkeypatch`` gets a new instance of it. If e.g. a test uses another module-scoped fixture defined like this: @pytest.fixture(scope="module") def myfix2(monkeypatch): mp.setattr(...) return some_value this would invoke the ``monkeypatch`` fixture function a second time, resulting in a new instance for use by the ``myfix2`` instance. The same logic could be applied to other fixtures like "tmpdir" or user-defined ones. Do you like this idea? Would you find it helpful for your test suites? There is one issue i am not sure about yet, however. Currently, when a test requires fixture A and B, and B requires C and C requires A, then the two "A" would be exactly the same object, independently of what which scopes are declared. If A=="tmpdir", then the test's tmpdir and C's tmpdir would be the same directory. I often don't find this desirable. If tmpdir would be an "each" scoped fixture, then C and the test would each receive a clean new tmpdir. If that is a backward-compat issue, we could introduce another name for the new "each" scoped tmpdir. I usually find myself working around the problem of a "tmpdir" shared by multiple different fixtures, though. cheers, holger

6 14

(new) pytest plugins / note to plugin authors
by holger krekel Oct. 11, 2013

Oct. 11, 2013

I just did three pytest plugin releases to get out a few minor fixes and improvements. see below. As far as i am aware most existing plugins (as well as the new releases here) should be compatible to pytest-2.4.2. Some plugins like pytest-capturelog have seen their latest release three years ago but remain functional. A good sign that the plugin system is reasonable i guess. If someone wants to help with organizing plugin compatibility tests and maintaining a status page, i'd really appreciate that. With over 50 existing plugins it's not easy to stay on top. The current pytest.org page is only a start: http://pytest.org/latest/plugins.html#extplugins and here is the full pypi list: https://pypi.python.org/pypi?%3Aaction=search&term=pytest-&submit=search there are also some github/bitbucket-only plugins, in addition. And finally a note to plugin authors: please be subscribed to pytest-dev and post new plugin releases here. best, holger pytest-xdist-1.9: - changed LICENSE to MIT - fix duplicate reported test ids with --looponfailing (thanks Jeremy Thurgood) - fix pytest issue41: re-run tests on all file changes, not just randomly select ones like .py/.c. - fix pytest issue347: slaves running on top of Python3.2 will set PYTHONDONTWRITEYBTECODE to 1 to avoid import concurrency bugs. pytest-xprocess-0.8: - support python3 (tested on linux/win32) - split out pytest independent process support into xprocess.py - add info.kill() method and some smaller refactoring - fix various windows related Popen / killing details - add tests pytest-pep8: - use pytest-2.4.2 node.add_marker() API for adding "pep8" marker

2 9

pytest-2.4: may better fixtures be with you
by holger krekel Oct. 4, 2013

Oct. 4, 2013

The just released pytest-2.4.0 brings many improvements and numerous bug fixes while remaining plugin- and test-suite compatible apart from a few supposedly very minor incompatibilities. See below for a full list of details. A few feature highlights: - new yield-style fixtures `pytest.yield_fixture <http://pytest.org/latest/yieldfixture.html>`_, allowing to use existing with-style context managers in fixture functions. - improved pdb support: ``import pdb ; pdb.set_trace()`` now works without requiring prior disabling of stdout/stderr capturing. Also the ``--pdb`` options works now on collection and internal errors and we introduced a new experimental hook for IDEs/plugins to intercept debugging: ``pytest_exception_interact(node, call, report)``. - shorter monkeypatch variant to allow specifying an import path as a target, for example: ``monkeypatch.setattr("requests.get", myfunc)`` - better unittest/nose compatibility: all teardown methods are now only called if the corresponding setup method succeeded. - integrate tab-completion on command line options if you have `argcomplete <http://pypi.python.org/pypi/argcomplete>`_ configured. - allow boolean expression directly with skipif/xfail if a "reason" is also specified. - a new hook ``pytest_load_initial_conftests`` allows plugins like `pytest-django <http://pypi.python.org/pypi/pytest-django>`_ to influence the environment before conftest files import ``django``. - reporting: color the last line red or green depending if failures/errors occured or everything passed. The documentation has been updated to accomodate the changes, see `http://pytest.org <http://pytest.org>`_ To install or upgrade pytest:: pip install -U pytest # or easy_install -U pytest **Many thanks to all who helped, including Floris Bruynooghe, Brianna Laugher, Andreas Pelme, Anthon van der Neut, Anatoly Bubenkoff, Vladimir Keleshev, Mathieu Agopian, Ronny Pfannschmidt, Christian Theunert and many others.** may passing tests be with you, holger krekel Changes between 2.3.5 and 2.4 ----------------------------------- known incompatibilities: - if calling --genscript from python2.7 or above, you only get a standalone script which works on python2.7 or above. Use Python2.6 to also get a python2.5 compatible version. - all xunit-style teardown methods (nose-style, pytest-style, unittest-style) will not be called if the corresponding setup method failed, see issue322 below. - the pytest_plugin_unregister hook wasn't ever properly called and there is no known implementation of the hook - so it got removed. - pytest.fixture-decorated functions cannot be generators (i.e. use yield) anymore. This change might be reversed in 2.4.1 if it causes unforeseen real-life issues. However, you can always write and return an inner function/generator and change the fixture consumer to iterate over the returned generator. This change was done in lieu of the new ``pytest.yield_fixture`` decorator, see below. new features: - experimentally introduce a new ``pytest.yield_fixture`` decorator which accepts exactly the same parameters as pytest.fixture but mandates a ``yield`` statement instead of a ``return statement`` from fixture functions. This allows direct integration with "with-style" context managers in fixture functions and generally avoids registering of finalization callbacks in favour of treating the "after-yield" as teardown code. Thanks Andreas Pelme, Vladimir Keleshev, Floris Bruynooghe, Ronny Pfannschmidt and many others for discussions. - allow boolean expression directly with skipif/xfail if a "reason" is also specified. Rework skipping documentation to recommend "condition as booleans" because it prevents surprises when importing markers between modules. Specifying conditions as strings will remain fully supported. - reporting: color the last line red or green depending if failures/errors occured or everything passed. thanks Christian Theunert. - make "import pdb ; pdb.set_trace()" work natively wrt capturing (no "-s" needed anymore), making ``pytest.set_trace()`` a mere shortcut. - fix issue181: --pdb now also works on collect errors (and on internal errors) . This was implemented by a slight internal refactoring and the introduction of a new hook ``pytest_exception_interact`` hook (see next item). - fix issue341: introduce new experimental hook for IDEs/terminals to intercept debugging: ``pytest_exception_interact(node, call, report)``. - new monkeypatch.setattr() variant to provide a shorter invocation for patching out classes/functions from modules: monkeypatch.setattr("requests.get", myfunc) will replace the "get" function of the "requests" module with ``myfunc``. - fix issue322: tearDownClass is not run if setUpClass failed. Thanks Mathieu Agopian for the initial fix. Also make all of pytest/nose finalizer mimick the same generic behaviour: if a setupX exists and fails, don't run teardownX. This internally introduces a new method "node.addfinalizer()" helper which can only be called during the setup phase of a node. - simplify pytest.mark.parametrize() signature: allow to pass a CSV-separated string to specify argnames. For example: ``pytest.mark.parametrize("input,expected", [(1,2), (2,3)])`` works as well as the previous: ``pytest.mark.parametrize(("input", "expected"), ...)``. - add support for setUpModule/tearDownModule detection, thanks Brian Okken. - integrate tab-completion on options through use of "argcomplete". Thanks Anthon van der Neut for the PR. - change option names to be hyphen-separated long options but keep the old spelling backward compatible. py.test -h will only show the hyphenated version, for example "--collect-only" but "--collectonly" will remain valid as well (for backward-compat reasons). Many thanks to Anthon van der Neut for the implementation and to Hynek Schlawack for pushing us. - fix issue 308 - allow to mark/xfail/skip individual parameter sets when parametrizing. Thanks Brianna Laugher. - call new experimental pytest_load_initial_conftests hook to allow 3rd party plugins to do something before a conftest is loaded. Bug fixes: - fix issue358 - capturing options are now parsed more properly by using a new parser.parse_known_args method. - pytest now uses argparse instead of optparse (thanks Anthon) which means that "argparse" is added as a dependency if installing into python2.6 environments or below. - fix issue333: fix a case of bad unittest/pytest hook interaction. - PR27: correctly handle nose.SkipTest during collection. Thanks Antonio Cuni, Ronny Pfannschmidt. - fix issue355: junitxml puts name="pytest" attribute to testsuite tag. - fix issue336: autouse fixture in plugins should work again. - fix issue279: improve object comparisons on assertion failure for standard datatypes and recognise collections.abc. Thanks to Brianna Laugher and Mathieu Agopian. - fix issue317: assertion rewriter support for the is_package method - fix issue335: document py.code.ExceptionInfo() object returned from pytest.raises(), thanks Mathieu Agopian. - remove implicit distribute_setup support from setup.py. - fix issue305: ignore any problems when writing pyc files. - SO-17664702: call fixture finalizers even if the fixture function partially failed (finalizers would not always be called before) - fix issue320 - fix class scope for fixtures when mixed with module-level functions. Thanks Anatloy Bubenkoff. - you can specify "-q" or "-qq" to get different levels of "quieter" reporting (thanks Katarzyna Jachim) - fix issue300 - Fix order of conftest loading when starting py.test in a subdirectory. - fix issue323 - sorting of many module-scoped arg parametrizations - make sessionfinish hooks execute with the same cwd-context as at session start (helps fix plugin behaviour which write output files with relative path such as pytest-cov) - fix issue316 - properly reference collection hooks in docs - fix issue 306 - cleanup of -k/-m options to only match markers/test names/keywords respectively. Thanks Wouter van Ackooy. - improved doctest counting for doctests in python modules -- files without any doctest items will not show up anymore and doctest examples are counted as separate test items. thanks Danilo Bellini. - fix issue245 by depending on the released py-1.4.14 which fixes py.io.dupfile to work with files with no mode. Thanks Jason R. Coombs. - fix junitxml generation when test output contains control characters, addressing issue267, thanks Jaap Broekhuizen - fix issue338: honor --tb style for setup/teardown errors as well. Thanks Maho. - fix issue307 - use yaml.safe_load in example, thanks Mark Eichin. - better parametrize error messages, thanks Brianna Laugher - pytest_terminal_summary(terminalreporter) hooks can now use ".section(title)" and ".line(msg)" methods to print extra information at the end of a test run.

1 2

please test pytest-2.4.0 release candidate
by holger krekel Oct. 1, 2013

Oct. 1, 2013

Hi everybody, could you help to verify that pytest-2.4 will continue to work for your test suite and/or plugins? Please report back any issues here or on #pylib freenode. Release is scheduled tomorrow (tuesday) noon (GMT) so please hurry :) The repo is now pretty much in shape for a release and you can install a packaged version like this:: pip install -i http://devpi.net/hpk/dev/+simple/ -U pytest this should give you pytest-2.4.dev13 (see "--version" option). Below is the long changelog in case you are curious. Latest two changes: - the decorator yield_fixture is now there. See http://pytest.org/dev/yieldfixture.html - i did a refactoring of how pytest startup is performed, fixing issue358 and allowing pytest-django and other plugins to perform actions before the first conftest file is loaded. Andreas Pelme already adapted pytest-django to utilize the improvements (mainly you can then use ``--ds=...`` setting and have your conftest.py files import django) and it seems to fix issue359. Apart from some known incompatibilities (see changelog below) pytest-2.4 aims to be a drop-in replacement for 2.3.5. cheers, holger Changes between 2.3.5 and 2.4.DEV ----------------------------------- known incompatibilities: - if calling --genscript from python2.7 or above, you only get a standalone script which works on python2.7 or above. Use Python2.6 to also get a python2.5 compatible version. - all xunit-style teardown methods (nose-style, pytest-style, unittest-style) will not be called if the corresponding setup method failed, see issue322 below. - the pytest_plugin_unregister hook wasn't ever properly called and there is no known implementation of the hook - so it got removed. - pytest.fixture-decorated functions cannot be generators (i.e. use yield) anymore. This change might be reversed in 2.4.1 if it causes unforeseen real-life issues. However, you can always write and return an inner function/generator and change the fixture consumer to iterate over the returned generator. This change was done in lieu of the new ``pytest.yield_fixture`` decorator, see below. new features: - experimentally introduce a new ``pytest.yield_fixture`` decorator which accepts exactly the same parameters as pytest.fixture but mandates a ``yield`` statement instead of a ``return statement`` from fixture functions. This allows direct integration with "with-style" context managers in fixture functions and generally avoids registering of finalization callbacks in favour of treating the "after-yield" as teardown code. Thanks Andreas Pelme, Vladimir Keleshev, Floris Bruynooghe, Ronny Pfannschmidt and many others for discussions. - allow boolean expression directly with skipif/xfail if a "reason" is also specified. Rework skipping documentation to recommend "condition as booleans" because it prevents surprises when importing markers between modules. Specifying conditions as strings will remain fully supported. - reporting: color the last line red or green depending if failures/errors occured or everything passed. thanks Christian Theunert. - make "import pdb ; pdb.set_trace()" work natively wrt capturing (no "-s" needed anymore), making ``pytest.set_trace()`` a mere shortcut. - fix issue181: --pdb now also works on collect errors (and on internal errors) . This was implemented by a slight internal refactoring and the introduction of a new hook ``pytest_exception_interact`` hook (see next item). - fix issue341: introduce new experimental hook for IDEs/terminals to intercept debugging: ``pytest_exception_interact(node, call, report)``. - new monkeypatch.setattr() variant to provide a shorter invocation for patching out classes/functions from modules: monkeypatch.setattr("requests.get", myfunc) will replace the "get" function of the "requests" module with ``myfunc``. - fix issue322: tearDownClass is not run if setUpClass failed. Thanks Mathieu Agopian for the initial fix. Also make all of pytest/nose finalizer mimick the same generic behaviour: if a setupX exists and fails, don't run teardownX. This internally introduces a new method "node.addfinalizer()" helper which can only be called during the setup phase of a node. - simplify pytest.mark.parametrize() signature: allow to pass a CSV-separated string to specify argnames. For example: ``pytest.mark.parametrize("input,expected", [(1,2), (2,3)])`` works as well as the previous: ``pytest.mark.parametrize(("input", "expected"), ...)``. - add support for setUpModule/tearDownModule detection, thanks Brian Okken. - integrate tab-completion on options through use of "argcomplete". Thanks Anthon van der Neut for the PR. - change option names to be hyphen-separated long options but keep the old spelling backward compatible. py.test -h will only show the hyphenated version, for example "--collect-only" but "--collectonly" will remain valid as well (for backward-compat reasons). Many thanks to Anthon van der Neut for the implementation and to Hynek Schlawack for pushing us. - fix issue 308 - allow to mark/xfail/skip individual parameter sets when parametrizing. Thanks Brianna Laugher. - call new experimental pytest_load_initial_conftests hook to allow 3rd party plugins to do something before a conftest is loaded. Bug fixes: - fix issue358 - capturing options are now parsed more properly by using a new parser.parse_known_args method. - pytest now uses argparse instead of optparse (thanks Anthon) which means that "argparse" is added as a dependency if installing into python2.6 environments or below. - fix issue333: fix a case of bad unittest/pytest hook interaction. - PR27: correctly handle nose.SkipTest during collection. Thanks Antonio Cuni, Ronny Pfannschmidt. - fix issue355: junitxml puts name="pytest" attribute to testsuite tag. - fix issue336: autouse fixture in plugins should work again. - fix issue279: improve object comparisons on assertion failure for standard datatypes and recognise collections.abc. Thanks to Brianna Laugher and Mathieu Agopian. - fix issue317: assertion rewriter support for the is_package method - fix issue335: document py.code.ExceptionInfo() object returned from pytest.raises(), thanks Mathieu Agopian. - remove implicit distribute_setup support from setup.py. - fix issue305: ignore any problems when writing pyc files. - SO-17664702: call fixture finalizers even if the fixture function partially failed (finalizers would not always be called before) - fix issue320 - fix class scope for fixtures when mixed with module-level functions. Thanks Anatloy Bubenkoff. - you can specify "-q" or "-qq" to get different levels of "quieter" reporting (thanks Katarzyna Jachim) - fix issue300 - Fix order of conftest loading when starting py.test in a subdirectory. - fix issue323 - sorting of many module-scoped arg parametrizations - make sessionfinish hooks execute with the same cwd-context as at session start (helps fix plugin behaviour which write output files with relative path such as pytest-cov) - fix issue316 - properly reference collection hooks in docs - fix issue 306 - cleanup of -k/-m options to only match markers/test names/keywords respectively. Thanks Wouter van Ackooy. - improved doctest counting for doctests in python modules -- files without any doctest items will not show up anymore and doctest examples are counted as separate test items. thanks Danilo Bellini. - fix issue245 by depending on the released py-1.4.14 which fixes py.io.dupfile to work with files with no mode. Thanks Jason R. Coombs. - fix junitxml generation when test output contains control characters, addressing issue267, thanks Jaap Broekhuizen - fix issue338: honor --tb style for setup/teardown errors as well. Thanks Maho. - fix issue307 - use yaml.safe_load in example, thanks Mark Eichin. - better parametrize error messages, thanks Brianna Laugher - pytest_terminal_summary(terminalreporter) hooks can now use ".section(title)" and ".line(msg)" methods to print extra information at the end of a test run.

3 4