Mailman 3 September 2013 - pytest-dev

please test pytest-2.4.0 release candidate
by holger krekel Oct. 1, 2013

Oct. 1, 2013

Hi everybody, could you help to verify that pytest-2.4 will continue to work for your test suite and/or plugins? Please report back any issues here or on #pylib freenode. Release is scheduled tomorrow (tuesday) noon (GMT) so please hurry :) The repo is now pretty much in shape for a release and you can install a packaged version like this:: pip install -i http://devpi.net/hpk/dev/+simple/ -U pytest this should give you pytest-2.4.dev13 (see "--version" option). Below is the long changelog in case you are curious. Latest two changes: - the decorator yield_fixture is now there. See http://pytest.org/dev/yieldfixture.html - i did a refactoring of how pytest startup is performed, fixing issue358 and allowing pytest-django and other plugins to perform actions before the first conftest file is loaded. Andreas Pelme already adapted pytest-django to utilize the improvements (mainly you can then use ``--ds=...`` setting and have your conftest.py files import django) and it seems to fix issue359. Apart from some known incompatibilities (see changelog below) pytest-2.4 aims to be a drop-in replacement for 2.3.5. cheers, holger Changes between 2.3.5 and 2.4.DEV ----------------------------------- known incompatibilities: - if calling --genscript from python2.7 or above, you only get a standalone script which works on python2.7 or above. Use Python2.6 to also get a python2.5 compatible version. - all xunit-style teardown methods (nose-style, pytest-style, unittest-style) will not be called if the corresponding setup method failed, see issue322 below. - the pytest_plugin_unregister hook wasn't ever properly called and there is no known implementation of the hook - so it got removed. - pytest.fixture-decorated functions cannot be generators (i.e. use yield) anymore. This change might be reversed in 2.4.1 if it causes unforeseen real-life issues. However, you can always write and return an inner function/generator and change the fixture consumer to iterate over the returned generator. This change was done in lieu of the new ``pytest.yield_fixture`` decorator, see below. new features: - experimentally introduce a new ``pytest.yield_fixture`` decorator which accepts exactly the same parameters as pytest.fixture but mandates a ``yield`` statement instead of a ``return statement`` from fixture functions. This allows direct integration with "with-style" context managers in fixture functions and generally avoids registering of finalization callbacks in favour of treating the "after-yield" as teardown code. Thanks Andreas Pelme, Vladimir Keleshev, Floris Bruynooghe, Ronny Pfannschmidt and many others for discussions. - allow boolean expression directly with skipif/xfail if a "reason" is also specified. Rework skipping documentation to recommend "condition as booleans" because it prevents surprises when importing markers between modules. Specifying conditions as strings will remain fully supported. - reporting: color the last line red or green depending if failures/errors occured or everything passed. thanks Christian Theunert. - make "import pdb ; pdb.set_trace()" work natively wrt capturing (no "-s" needed anymore), making ``pytest.set_trace()`` a mere shortcut. - fix issue181: --pdb now also works on collect errors (and on internal errors) . This was implemented by a slight internal refactoring and the introduction of a new hook ``pytest_exception_interact`` hook (see next item). - fix issue341: introduce new experimental hook for IDEs/terminals to intercept debugging: ``pytest_exception_interact(node, call, report)``. - new monkeypatch.setattr() variant to provide a shorter invocation for patching out classes/functions from modules: monkeypatch.setattr("requests.get", myfunc) will replace the "get" function of the "requests" module with ``myfunc``. - fix issue322: tearDownClass is not run if setUpClass failed. Thanks Mathieu Agopian for the initial fix. Also make all of pytest/nose finalizer mimick the same generic behaviour: if a setupX exists and fails, don't run teardownX. This internally introduces a new method "node.addfinalizer()" helper which can only be called during the setup phase of a node. - simplify pytest.mark.parametrize() signature: allow to pass a CSV-separated string to specify argnames. For example: ``pytest.mark.parametrize("input,expected", [(1,2), (2,3)])`` works as well as the previous: ``pytest.mark.parametrize(("input", "expected"), ...)``. - add support for setUpModule/tearDownModule detection, thanks Brian Okken. - integrate tab-completion on options through use of "argcomplete". Thanks Anthon van der Neut for the PR. - change option names to be hyphen-separated long options but keep the old spelling backward compatible. py.test -h will only show the hyphenated version, for example "--collect-only" but "--collectonly" will remain valid as well (for backward-compat reasons). Many thanks to Anthon van der Neut for the implementation and to Hynek Schlawack for pushing us. - fix issue 308 - allow to mark/xfail/skip individual parameter sets when parametrizing. Thanks Brianna Laugher. - call new experimental pytest_load_initial_conftests hook to allow 3rd party plugins to do something before a conftest is loaded. Bug fixes: - fix issue358 - capturing options are now parsed more properly by using a new parser.parse_known_args method. - pytest now uses argparse instead of optparse (thanks Anthon) which means that "argparse" is added as a dependency if installing into python2.6 environments or below. - fix issue333: fix a case of bad unittest/pytest hook interaction. - PR27: correctly handle nose.SkipTest during collection. Thanks Antonio Cuni, Ronny Pfannschmidt. - fix issue355: junitxml puts name="pytest" attribute to testsuite tag. - fix issue336: autouse fixture in plugins should work again. - fix issue279: improve object comparisons on assertion failure for standard datatypes and recognise collections.abc. Thanks to Brianna Laugher and Mathieu Agopian. - fix issue317: assertion rewriter support for the is_package method - fix issue335: document py.code.ExceptionInfo() object returned from pytest.raises(), thanks Mathieu Agopian. - remove implicit distribute_setup support from setup.py. - fix issue305: ignore any problems when writing pyc files. - SO-17664702: call fixture finalizers even if the fixture function partially failed (finalizers would not always be called before) - fix issue320 - fix class scope for fixtures when mixed with module-level functions. Thanks Anatloy Bubenkoff. - you can specify "-q" or "-qq" to get different levels of "quieter" reporting (thanks Katarzyna Jachim) - fix issue300 - Fix order of conftest loading when starting py.test in a subdirectory. - fix issue323 - sorting of many module-scoped arg parametrizations - make sessionfinish hooks execute with the same cwd-context as at session start (helps fix plugin behaviour which write output files with relative path such as pytest-cov) - fix issue316 - properly reference collection hooks in docs - fix issue 306 - cleanup of -k/-m options to only match markers/test names/keywords respectively. Thanks Wouter van Ackooy. - improved doctest counting for doctests in python modules -- files without any doctest items will not show up anymore and doctest examples are counted as separate test items. thanks Danilo Bellini. - fix issue245 by depending on the released py-1.4.14 which fixes py.io.dupfile to work with files with no mode. Thanks Jason R. Coombs. - fix junitxml generation when test output contains control characters, addressing issue267, thanks Jaap Broekhuizen - fix issue338: honor --tb style for setup/teardown errors as well. Thanks Maho. - fix issue307 - use yaml.safe_load in example, thanks Mark Eichin. - better parametrize error messages, thanks Brianna Laugher - pytest_terminal_summary(terminalreporter) hooks can now use ".section(title)" and ".line(msg)" methods to print extra information at the end of a test run.

3 4

Case for @fixture to support yield-style fixtures directly
by Vladimir Keleshev Sept. 29, 2013

Sept. 29, 2013

I would like to make a case for making yield-style fixtures to be supported directly by the @fixture decorator. But before that I want to remind some of you on what I am talking about. Yield-sytle fixtures are planned to be added in pytest 2.4, but their exact API is under discussion. Let me remind you that yield-style fixture looks like: @fixture def empty_db(): db = DB() # Setup code. yield db db.clear().close() # Teardown code. Now contrast with pytest 2.3 fixtures: @fixture def empty_db(request): db = DB() request.addfinalizer(lambda: db.clear().close()) return db Both create an instance of a hypothetical database, return it as a fixture, then perform cleanup. For a similar real-life example take a look here: http://github.com/halst/urlbeat/blob/master/test_urlbeat.py Current docs draft (http://pytest.org/dev/yieldfixture.html) states some advantages and disadvantages which I want to comment on. 1.1 Streamlined control flow. Callbacks are known to (visually) break the order of control flow, which contributes to their poor readability. At the same time, having tear-down code after `yield` might seem unusuall at first, but I think that such fixtures are so readable to the point of being called beautiful. You can read them from top to bottom: first, set-up code, then the object is yielded, then the tear-down follows. 1.2 No need for `request` argument. In my experience with pytest 2.3 and before, the `request` argument of fixtures is used for defining finalizers in about 90% of the cases. Other uses are relatively rare. Yield-sytle fixtures allow to simplify this by allowing to drop that argument. 1.3. Simple fixtures that use context managers. This is important; yield-style fixtures allow to use context managers (think `with` statement) inside fixtures. For example, if a `DB` class is designed to be a context manager, you would write the following yield-style fixture: @fixture def emtpy_db(): with DB() as db: yield db However, with current pytest 2.3 fixtures that would approximately be: @fixture def empty_db(request): db = DB.__enter__() request.addfinalizer( lambda: db.__exit__(None, None, None)) return db What I want to stress is that context managers were specifically introduced for this purpose: handle resources that need some set-up and tear-down. Almost all such resources now support them. That is one of the reasons, I think, that yield-style fixtures are such a huge improvement. * * * To sum up the advantages: you can argue about readability, but you can't argue that support for context-managers is an extremely important use-case. Both fixtures and context managers are made primarily to set-up and tear-down resources. Increasing numbers of API support context managers, and I'm sure that most of existing fixtures (that use `.close()` and `.clear()` and similar methods) could be rewritten more elegantly (and more uniformly) to use yield-style fixtures with context managers. * * * Now, to comment on disadvantages: 2.1. Existing fixtures might use `yield` for ad-hoc parametrization. That means that yield-sytle fixtures might break existing code that uses generators for parametrization like this: @fixture def all_backends(request): for Backend in [MemoryBackend, BinaryBackend, CSVBackend]: backend = Backend() yield backend request.addfinalizer(backend.clear) def test_backends(all_backends): for backend in all_backends: assert backend.put('key', 42).get('key') == 42 For one, I have never seen such code, and for two, the same could be achieved more elegantly with parametrization API: @fixture(params=[MemroyBackend, BinaryBackend, CSVBackend]) def backend(request): backend = request.param() request.addfinilizer(backend.clear) return backend def test_backends(backend): assert backend.put('key', 42).get('key') == 42 Plus, as of pytest 2.4.0.dev11, the user of such ad-hoc generator fixture will see a very clear error message: fixture function backends has more than one `yield` Which could elaborate even more if necessary: Fixture function `backends` has more than one `yield`. Since pytest 2.4 fixtures with `yield` statement are treated differently: <link>. You might want to use parametrization API instead: <link>. 2.2 Yield-style fixtures introduce more that one way to do fixtures (together with current return-style fixtures) Yield-style fixtures present us with a superset of functionality of what return-style fixtures can achieve. I would even like to say that except for backwards-compatibility and user habit concerns, there is no need for return-style fixtures at all, if yield-style is available. So I would even argue that yield-style fixtures should take the main place in fixture documentation, and return-style fixtures mentioned for backwards compatibility (as funcargs are). * * * To sum up, the only real disadvantage of yield-style fixtures is that they can break a single hypothetical use-case, which is either rare or nonexistent, and for which a clear error-message is provided. On the other hand, they allow to use context managers for finalization, which were specifically designed to hanldle this use-case elegantly. I'm pretty sure that 100% of fixtures that use `request.addfinalizer` will benefit from yield-style fixtures, and that most of them will be able to use context managers for finalization. Just look at all the evolution steps. 3.1 Past: def pytest_funcarg__db(request): db = DB() request.addfinalizer(db.clear) return db 3.2 Present: @fixture def db(request): db = DB() request.addfinalizer(db.clear) return db 3.3 Future: @fixture def db(): db = DB() yield db db.clear() Or even better: @fixture def db(): with DB() as db: yield db So, do you agree that this is the future we want to be in? * * * There are a certain challenges that we need to solve then. 4.1 Backwards-compatibility (which I already mentioned). 4.2 There is a high chance (in my experience) of mistyping `return` for `yield` like: @fixture def db(): db = DB() return db db.clear() Even worse, this fixture could still appear to work, until some later time, when they system runs out of file descriptors, for example. Here are the 3 solutions discussed: 5.1 Extra argument to `fixture` to designate yield-style: @fixture(yield_context=True) def db(): db = DB() yield db db.clear() 5.2 Different fixture decorator to designate yield-sytle: @yield_fixture def db(): db = DB() yield db db.clear() 5.3 A configuration option, say, in pytest.ini that will switch bettween return-style and yield-style. What I would like to propose is: 6.1 Keep single @fixture decorator. 6.2 Give user a clear error-message in case fixture yields more than one time (done). 6.3 Using simple static-analysis, warn user when there are statements after `return`, to get rid of return/yield typos. 4. Document yield-style as the preferable way of finalization. All of that keeps the API simple (no new decorators, arguments, configuration options), warns user in case of every known caveat, and finally, provides us the beautiful yield-style fixtures by default. —Vladimir Keleshev

3 4

monkeypatch.setattr versus replace
by holger krekel Sept. 17, 2013

Sept. 17, 2013

Hi all, this was discussed before on IRC, but here is an update and a question. currently pytest-trunk implements a convenient new monkeypatch.replace method: # example usage monkeypatch.replace("os.path.abspath", lambda x: "/abc") This effectively performs: import os.path monkeypatch.setattr(os.path, "abspath", lambda x: "/abc") However, i just wanted to use it and couldn't remember the "replace" name and don't find it very telling anymore. I often do something like this:: monkeypatch.setattr("os.path.abspath", lambda x: "/abc") which currently ends up as an error. Would it be too awkward to just make the latter work and forget about introducing a new "replace" method name? IMO it's rather intuitive to just make setattr accept the new signature, although the docstring will need a bit of explaining. cheers, holger

4 7

Re: [pytest-dev] monkeypatch.setattr versus replace
by holger krekel Sept. 16, 2013

Sept. 16, 2013

On Fri, Sep 13, 2013 at 13:12 +0200, Anthon van der Neut wrote: > On 2013-09-13 12:33, holger krekel wrote: > >Hi all, > > > >this was discussed before on IRC, but here is an update and a question. > >currently pytest-trunk implements a convenient new monkeypatch.replace method: > > > > # example usage > > monkeypatch.replace("os.path.abspath", lambda x: "/abc") > > > >This effectively performs: > > > > import os.path > > monkeypatch.setattr(os.path, "abspath", lambda x: "/abc") > > > >However, i just wanted to use it and couldn't remember the "replace" > >name and don't find it very telling anymore. I often do something > >like this:: > > > > monkeypatch.setattr("os.path.abspath", lambda x: "/abc") > > As long as the old 3 parameter setattr is not going to be away, that > seems fine. > Depending on the length of the string I would probably still use the > 3 parameter version to get the <TAB> completion on the first > argument. > Unfortunately just writing > > monkeypatch.setattr(os.path.abspath, lambda x: "/abc") It's actually possible to make this work in many common cases, but is bordering on the magic side. holger

2 1

py.test doesn't detect/warn on duplicate tests with parametrize
by Brianna Laugher Sept. 6, 2013

Sept. 6, 2013

Hi, In our distributed tests we have been having intermittent failures on one particular parametrized test, with the failure always being on the same test case - failing in pytest_pyfunc_call in the look-up of a funcarg. This funcarg is used in a very large number of our tests (ie otherwise seems reliable) and has function scope. I was just looking into it (traceback below) and I suddenly realised that the case was duplicated. Our test has 135 entries in parametrize so obviously someone (probably me!) just copy-pasted and accidentally left one line the same. It seems to me it might be appropriate for py.test to warn about this, or even give a collection error? What do people think? Secondly does it seem like this might somehow be related to the intermittent test failure? It is baffling me. thanks, Brianna ../pytest-2.3.5-py2.7.egg/_pytest/runner.py:129: in __init__ > self.result = func() ../pytest-2.3.5-py2.7.egg/_pytest/runner.py:116: in <lambda> > return CallInfo(lambda: ihook(item=item, **kwds), when=when) ../pytest-2.3.5-py2.7.egg/_pytest/main.py:159: in call_matching_hooks > return hookmethod.pcall(plugins, **kwargs) ../pytest-2.3.5-py2.7.egg/_pytest/core.py:445: in pcall > return self._docall(methods, kwargs) ../pytest-2.3.5-py2.7.egg/_pytest/core.py:452: in _docall > res = mc.execute() ../pytest-2.3.5-py2.7.egg/_pytest/core.py:370: in execute > res = method(**kwargs) ../pytest-2.3.5-py2.7.egg/_pytest/runner.py:86: in pytest_runtest_call > item.runtest() ../pytest-2.3.5-py2.7.egg/_pytest/python.py:982: in runtest > self.ihook.pytest_pyfunc_call(pyfuncitem=self) ../pytest-2.3.5-py2.7.egg/_pytest/main.py:159: in call_matching_hooks > return hookmethod.pcall(plugins, **kwargs) ../pytest-2.3.5-py2.7.egg/_pytest/core.py:445: in pcall > return self._docall(methods, kwargs) ../pytest-2.3.5-py2.7.egg/_pytest/core.py:452: in _docall > res = mc.execute() ../pytest-2.3.5-py2.7.egg/_pytest/core.py:370: in execute > res = method(**kwargs) ../pytest-2.3.5-py2.7.egg/_pytest/python.py:152: in pytest_pyfunc_call > testargs[arg] = funcargs[arg] E KeyError: 'dbs' -- They've just been waiting in a mountain for the right moment: http://modernthings.org/

2 1

support for --pdb on collect/internal errors
by holger krekel Sept. 6, 2013

Sept. 6, 2013

Hi folks, if you are hacking hooks or otherwise debugging pytest and plugins you can now use "--pdb" when collect/internal errors appear. For that, I have introduced a new experimental hook "pytest_exception_interact" which could also be used by IDE integrators for their debugging views, e.g.: def pytest_exception_interact(call): excinfo = call.excinfo pydev.set_trace(excinfo.type, excinfo.value, excinfo.tb) when this hook is called, stdout/stderr capturing will be de-activated already. It could also be used to spawn up a http/socket server so that one could remotely connect to it for debugging sessions. This is the commit: https://bitbucket.org/hpk42/pytest/commits/1d3ebb313d68d4bcb3f1a2e575febf19… cheers, holger

1 0

How to exclude packages/directories from py.test?
by Yang Zhang Sept. 5, 2013

Sept. 5, 2013

You can use -k to filter out tests following a certain name pattern/predicate, but can you filter out tests under a certain package or directory? Didn't spot any relevant flags in the documentation but perhaps I'm missing something. Thanks.

2 3