From issues-reply at bitbucket.org Sun Jul 1 15:42:34 2012 From: issues-reply at bitbucket.org (Ronny Pfannschmidt) Date: Sun, 01 Jul 2012 13:42:34 -0000 Subject: [py-dev] [hpk42/pytest-cache] --lf and keyword selection cause misstakes (issue #1) Message-ID: <806360810060177e20368f4d7e2d40e9@bitbucket.org> --- you can reply above this line --- New issue 1: --lf and keyword selection cause misstakes https://bitbucket.org/hpk42/pytest-cache/issue/1/lf-and-keyword-selection-cause-misstakes Ronny Pfannschmidt: while testing i noticed, that if i try to run a subset of the last failures with -k, it considers the rest of the tests passed and runs all of the tests the next time instead it should remember those that where deselected but didnt run -- This is an issue notification from bitbucket.org. You are receiving this either because you are the owner of the issue, or you are following the issue. From holger at merlinux.eu Mon Jul 2 13:56:49 2012 From: holger at merlinux.eu (holger krekel) Date: Mon, 2 Jul 2012 11:56:49 +0000 Subject: [py-dev] RFC: V2 of the new resource setup/parametrization facilities In-Reply-To: <20120630112605.GA32749@laurie.devork.be> References: <20120629105523.GW11942@merlinux.eu> <20120630002304.GB26143@laurie.devork.be> <20120630080839.GY11942@merlinux.eu> <20120630112605.GA32749@laurie.devork.be> Message-ID: <20120702115649.GB11942@merlinux.eu> On Sat, Jun 30, 2012 at 12:26 +0100, Floris Bruynooghe wrote: > > On Sat, Jun 30, 2012 at 01:23 +0100, Floris Bruynooghe wrote: > > > On Fri, Jun 29, 2012 at 10:55:23AM +0000, holger krekel wrote: > > > > > > > > def setup_directory(db): > > > > # called when the first test in the directory tree is about > > > > to execute > > > > > > I think the naming of these functions break the py.test convention, > > > normally the only functions picked up from conftest.py start with > > > pytest_. I can certainly imagine a conftest.py or plugin already > > > having a setup_session function. These are new functions and do not > > > provide a compatibility API with other testing frameworks, so I think > > > they would be better named pytest_setup_session and > > > pytest_setup_directory. > > > > I think using pytest_* hooks also has consistency problems: > > > > * hooks cannot usually receive arbitrary funcargs > > This is why a signature with a request/node for these might be better:: > > def pytest_setup_session(session): > session.getresource('db') # or .getfuncargvalue()? > ... > > > * xUnit-style consistency: consider explaining the new functions > > to someone only knowing setup_module/ class etc. > > As I tried to say before, they do not come for xUnit so I don't think > this is too important. I think the consistency inside conftest.py is > more important. Well, pytest introduced setup_module/class and nose/unittest ported it. I consider setup_directory (or setup_session) to be xUnit consistent from a user perspective and maybe nose/unittest will also add it - i guess there are extensions there implementing something like this already. In generaly, if we go the route of making setup_X more powerful there probably is less need for pytest_runtest_setup calls. The only difference would be that runtest_setup is called in plugins/conftest.py whereas setup_function/method need to be defined around the actual test code. (TBH i wonder if setup_module/class/function/method could be allowed in conftest.py files as well - in many cases there are easier to handle than pytest_runtest_item(item) which does not even guarantee that the item is a python function - could be a PEP8 checker Item). > > I am wondering, however, do we even need a "setup_session"? setup_directory > > should usually be enough, i guess, and it's more unlikely people used > > that name already (and we could warn about setup_session in 2.X to > > reserve introducing it in 2.X+1). > > Maybe not, but if you don't provide setup_session (or > pytest_setup_session) then pytest_sessionstart will be used again > when someone thinks of a reason to use it. And that's what you wanted > to avoid. We definitely need to provide prominent examples for whole-session setup to avoid further usage of sessionstart or configure. > [...] > > If a setup-function has no body, then tests could just require it themselves > > and that'd be enough. If there is a need, we could introduce a marker for > > requiring funcarg-resources such that tests do not need to require it > > in their signature. > > I'm not sure what that would save, either the test function must > request the resource or must be marked to need the resource. If > anything the second takes more work. > > > As an aside however, one of my usecases for merged request/item > objects was so I could put setup in a session-wide scoped funcarg but > also automatically request this funcarg based on a mark:: > > def pytest_runtest_setup(item): > if 'needsdb' in item.keywords: # or a more explicit API > item.getresource('db') > > I understand that this will still be possible via:: > > def pytest_runtest_setup(item): > if 'needsdb' in item.keywords: > item.session.getresource('db') > > Or something similar to that. It'd probably be best if this requirement is know at collection time so --collectonly can present the complete picture. This could look like this:: def pytest_itemcollected(item): if 'needsdb' in item.keywords: item.applymarker(pytest.mark.needsresource("db")) Also, the above issue of requiring a global resource could be expressed like this:: def pytest_collection_finish(session): session.applymarker(pytest.mark.needsresource("db")) This should also in principle work well in case of a parametrized "db" so that all tests requiring "db" can be run multiple times. (I am not sure if the above already existing hooks fit well for that, however.) best, holger From issues-reply at bitbucket.org Mon Jul 2 16:06:23 2012 From: issues-reply at bitbucket.org (Ronny Pfannschmidt) Date: Mon, 02 Jul 2012 14:06:23 -0000 Subject: [py-dev] [hpk42/pytest-cache] --lf ignores collect failures (issue #2) Message-ID: <522b23612aef2277441451cb702dbff9@bitbucket.org> --- you can reply above this line --- New issue 2: --lf ignores collect failures https://bitbucket.org/hpk42/pytest-cache/issue/2/lf-ignores-collect-failures Ronny Pfannschmidt: it would be nice if it was aware of those as well, so it wont run all tests in such cases -- This is an issue notification from bitbucket.org. You are receiving this either because you are the owner of the issue, or you are following the issue. From carl at oddbird.net Fri Jul 6 01:35:49 2012 From: carl at oddbird.net (Carl Meyer) Date: Thu, 05 Jul 2012 17:35:49 -0600 Subject: [py-dev] RFC: V2 of the new resource setup/parametrization facilities In-Reply-To: <20120629105523.GW11942@merlinux.eu> References: <20120629105523.GW11942@merlinux.eu> Message-ID: <4FF624D5.3000703@oddbird.net> Hello Holger, On 06/29/2012 04:55 AM, holger krekel wrote: > I believe that the new resource parametrization facilities are a major > step forward - they should allow test writers to much more seldomly having > to resort to pytest_* hooks, and make access and working with parametrized > resources straight forward, irrespective of previous xUnit/pytest background. I think so too! Very excited about the more straightforward parametrization and cache-scope for resources. I'm of course fairly new to pytest and so not as familiar with history or internal implementation, but perhaps that new perspective is also useful. On the whole I think the proposal is excellent. I would also prefer to see pytest.mark.factory_scope and pytest.mark.factory_parametrize combined into a single decorator, but I don't like the name "factoryattr". For one thing, I find the usage of "factory" a bit confusing (there's no clarity about what sort of factory). And second, I would go even further and allow this decorator itself (even with no arguments) to mark a function as a funcarg, even if the factory function is not named according to the pytest_funcarg__foo convention. So for instance: pytest.mark.resource() def database(request): # ... (The decorator could also be "pytest.mark.funcarg()" if you're not ready to move towards the "resource" naming. I think "resource" is more intuitive than "funcarg" for newcomers, but I recognize the costs of shifting.) And then this decorator would of course also accept the "scope" and "parametrize" keyword arguments. To my mind, this would be a really nice unification and could become the canonical way to mark a test resource, with the naming convention becoming either a deprecated backwards-compatibility feature, or an approved and supported shortcut for the no-extra-parameters case, depending how you feel about it. (Personally I prefer explicit marking to naming conventions, but I realize this preference may not be in the spirit of pytest). Regarding the naming of "scope" and the dual scopes (lifecycle and visibility), personally I don't think it's a big issue to just use the name "scope"; does the name "scope" actually occur in the API related to the visibility variety? I find "cachescope" irritating and a bit misleading. If "scope" is overloaded, what about "lifetime"? Speaking of the lifetime of an instance of a test resource feels more natural to me than speaking of caching it. Carl From holger at merlinux.eu Fri Jul 6 11:20:20 2012 From: holger at merlinux.eu (holger krekel) Date: Fri, 6 Jul 2012 09:20:20 +0000 Subject: [py-dev] RFC: V2 of the new resource setup/parametrization facilities In-Reply-To: <4FF624D5.3000703@oddbird.net> References: <20120629105523.GW11942@merlinux.eu> <4FF624D5.3000703@oddbird.net> Message-ID: <20120706092020.GJ18336@merlinux.eu> Hi Carl, thanks. I agree on your points and they also connect to comments from Floris. On Thu, Jul 05, 2012 at 17:35 -0600, Carl Meyer wrote: > Hello Holger, > > On 06/29/2012 04:55 AM, holger krekel wrote: > > I believe that the new resource parametrization facilities are a major > > step forward - they should allow test writers to much more seldomly having > > to resort to pytest_* hooks, and make access and working with parametrized > > resources straight forward, irrespective of previous xUnit/pytest background. > > I think so too! Very excited about the more straightforward > parametrization and cache-scope for resources. > > I'm of course fairly new to pytest and so not as familiar with history > or internal implementation, but perhaps that new perspective is also > useful. On the whole I think the proposal is excellent. > > I would also prefer to see pytest.mark.factory_scope and > pytest.mark.factory_parametrize combined into a single decorator, but I > don't like the name "factoryattr". For one thing, I find the usage of > "factory" a bit confusing (there's no clarity about what sort of > factory). And second, I would go even further and allow this decorator > itself (even with no arguments) to mark a function as a funcarg, even if > the factory function is not named according to the pytest_funcarg__foo > convention. So for instance: > > pytest.mark.resource() > def database(request): > # ... > > (The decorator could also be "pytest.mark.funcarg()" if you're not ready > to move towards the "resource" naming. I think "resource" is more > intuitive than "funcarg" for newcomers, but I recognize the costs of > shifting.) I actually think that moving to "resource" and talking about "funcarg resource" make some sense. > > And then this decorator would of course also accept the "scope" and > "parametrize" keyword arguments. To my mind, this would be a really nice > unification and could become the canonical way to mark a test resource, > with the naming convention becoming either a deprecated > backwards-compatibility feature, or an approved and supported shortcut > for the no-extra-parameters case, depending how you feel about it. > (Personally I prefer explicit marking to naming conventions, but I > realize this preference may not be in the spirit of pytest). > > Regarding the naming of "scope" and the dual scopes (lifecycle and > visibility), personally I don't think it's a big issue to just use the > name "scope"; does the name "scope" actually occur in the API related to > the visibility variety? > > I find "cachescope" irritating and a bit misleading. If "scope" is > overloaded, what about "lifetime"? Speaking of the lifetime of an > instance of a test resource feels more natural to me than speaking of > caching it. Ok, so i'd go for: @pytest.mark.resource(scope=..., parametrize=...) and also allow using it with no arguments, releasing the requirement for special naming (which originates from looking at Django FWIW :) Still pondering if "resourcefactory" would be more suitable, however. best, holger > > Carl > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > From sontek at gmail.com Sat Jul 7 04:31:45 2012 From: sontek at gmail.com (John Anderson) Date: Fri, 6 Jul 2012 21:31:45 -0500 Subject: [py-dev] INTERNAL ERROR doesn't give a good exit code? In-Reply-To: References: Message-ID: I'm running my builds on travis-ci and I had some bad imports and it threw an INTERNAL ERROR on py.test, but gave an exit code 0 so the build doesn't fail.. Any way to fix this? ============================= test session starts ============================== 626platform linux2 -- Python 2.7.2 -- pytest-2.2.4 627INTERNALERROR> Traceback (most recent call last): 628INTERNALERROR> File "/home/vagrant/virtualenv/python2.7/local/lib/python2.7/site-packages/_pytest/main.py", line 72, in wrap_session 629INTERNALERROR> config.hook.pytest_sessionstart(session=session) 630INTERNALERROR> File "/home/vagrant/virtualenv/python2.7/local/lib/python2.7/site-packages/_pytest/core.py", line 421, in __call__ 631INTERNALERROR> return self._docall(methods, kwargs) 632INTERNALERROR> File "/home/vagrant/virtualenv/python2.7/local/lib/python2.7/site-packages/_pytest/core.py", line 432, in _docall 633INTERNALERROR> res = mc.execute() 634INTERNALERROR> File "/home/vagrant/virtualenv/python2.7/local/lib/python2.7/site-packages/_pytest/core.py", line 350, in execute 635INTERNALERROR> res = method(**kwargs) 636INTERNALERROR> File "/home/vagrant/builds/sontek/hiero/conftest.py", line 10, in pytest_sessionstart 637INTERNALERROR> from hiero.tests.models import Base 638INTERNALERROR> File "/home/vagrant/builds/sontek/hiero/hiero/__init__.py", line 2, in 639INTERNALERROR> from hem.config import get_class_from_config 640INTERNALERROR> ImportError: No module named hem.config 641 642Done. Build script exited with: 0 From sontek at gmail.com Sat Jul 7 05:30:23 2012 From: sontek at gmail.com (John Anderson) Date: Fri, 6 Jul 2012 22:30:23 -0500 Subject: [py-dev] INTERNAL ERROR doesn't give a good exit code? In-Reply-To: References: Message-ID: Turns out it was the documented setup.py PyTestCommand I found: class PyTest(TestCommand): def finalize_options(self): TestCommand.finalize_options(self) self.test_args = [] self.test_suite = True def run_tests(self): #import here, cause outside the eggs aren't loaded import pytest pytest.main(self.test_args) Changed the last line to result = pytest.main(self.test_args) sys.exit(result) and everything is working. On Fri, Jul 6, 2012 at 9:31 PM, John Anderson wrote: > I'm running my builds on travis-ci and I had some bad imports and it > threw an INTERNAL ERROR on py.test, but gave an exit code 0 so the > build doesn't fail.. Any way to fix this? > > > ============================= test session starts ============================== > > 626platform linux2 -- Python 2.7.2 -- pytest-2.2.4 > > 627INTERNALERROR> Traceback (most recent call last): > > 628INTERNALERROR> File > "/home/vagrant/virtualenv/python2.7/local/lib/python2.7/site-packages/_pytest/main.py", > line 72, in wrap_session > > 629INTERNALERROR> config.hook.pytest_sessionstart(session=session) > > 630INTERNALERROR> File > "/home/vagrant/virtualenv/python2.7/local/lib/python2.7/site-packages/_pytest/core.py", > line 421, in __call__ > > 631INTERNALERROR> return self._docall(methods, kwargs) > > 632INTERNALERROR> File > "/home/vagrant/virtualenv/python2.7/local/lib/python2.7/site-packages/_pytest/core.py", > line 432, in _docall > > 633INTERNALERROR> res = mc.execute() > > 634INTERNALERROR> File > "/home/vagrant/virtualenv/python2.7/local/lib/python2.7/site-packages/_pytest/core.py", > line 350, in execute > > 635INTERNALERROR> res = method(**kwargs) > > 636INTERNALERROR> File > "/home/vagrant/builds/sontek/hiero/conftest.py", line 10, in > pytest_sessionstart > > 637INTERNALERROR> from hiero.tests.models import Base > > 638INTERNALERROR> File > "/home/vagrant/builds/sontek/hiero/hiero/__init__.py", line 2, in > > > 639INTERNALERROR> from hem.config import get_class_from_config > > 640INTERNALERROR> ImportError: No module named hem.config > > 641 > > 642Done. Build script exited with: 0 From holger at merlinux.eu Sat Jul 7 07:41:53 2012 From: holger at merlinux.eu (holger krekel) Date: Sat, 7 Jul 2012 05:41:53 +0000 Subject: [py-dev] INTERNAL ERROR doesn't give a good exit code? In-Reply-To: References: Message-ID: <20120707054153.GK18336@merlinux.eu> Hi John, thanks for fixing it - i committed it to the trunk docs. best, holger On Fri, Jul 06, 2012 at 22:30 -0500, John Anderson wrote: > Turns out it was the documented setup.py PyTestCommand I found: > > class PyTest(TestCommand): > def finalize_options(self): > TestCommand.finalize_options(self) > self.test_args = [] > self.test_suite = True > def run_tests(self): > #import here, cause outside the eggs aren't loaded > import pytest > pytest.main(self.test_args) > > Changed the last line to > result = pytest.main(self.test_args) > sys.exit(result) > > and everything is working. > > > On Fri, Jul 6, 2012 at 9:31 PM, John Anderson wrote: > > I'm running my builds on travis-ci and I had some bad imports and it > > threw an INTERNAL ERROR on py.test, but gave an exit code 0 so the > > build doesn't fail.. Any way to fix this? > > > > > > ============================= test session starts ============================== > > > > 626platform linux2 -- Python 2.7.2 -- pytest-2.2.4 > > > > 627INTERNALERROR> Traceback (most recent call last): > > > > 628INTERNALERROR> File > > "/home/vagrant/virtualenv/python2.7/local/lib/python2.7/site-packages/_pytest/main.py", > > line 72, in wrap_session > > > > 629INTERNALERROR> config.hook.pytest_sessionstart(session=session) > > > > 630INTERNALERROR> File > > "/home/vagrant/virtualenv/python2.7/local/lib/python2.7/site-packages/_pytest/core.py", > > line 421, in __call__ > > > > 631INTERNALERROR> return self._docall(methods, kwargs) > > > > 632INTERNALERROR> File > > "/home/vagrant/virtualenv/python2.7/local/lib/python2.7/site-packages/_pytest/core.py", > > line 432, in _docall > > > > 633INTERNALERROR> res = mc.execute() > > > > 634INTERNALERROR> File > > "/home/vagrant/virtualenv/python2.7/local/lib/python2.7/site-packages/_pytest/core.py", > > line 350, in execute > > > > 635INTERNALERROR> res = method(**kwargs) > > > > 636INTERNALERROR> File > > "/home/vagrant/builds/sontek/hiero/conftest.py", line 10, in > > pytest_sessionstart > > > > 637INTERNALERROR> from hiero.tests.models import Base > > > > 638INTERNALERROR> File > > "/home/vagrant/builds/sontek/hiero/hiero/__init__.py", line 2, in > > > > > > 639INTERNALERROR> from hem.config import get_class_from_config > > > > 640INTERNALERROR> ImportError: No module named hem.config > > > > 641 > > > > 642Done. Build script exited with: 0 > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev > From Ronny.Pfannschmidt at gmx.de Sat Jul 7 12:37:34 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Sat, 07 Jul 2012 12:37:34 +0200 Subject: [py-dev] [RFC] tox combinatorial environments V0.1 Message-ID: <4FF8116E.2000000@gmx.de> Hi all, After some recent thought I think I grasped the beginning of a good base idea for combinatoric tox envs Various users including myself have a need for better expression of more combinatoric dimensions in tox testenv listings example ranges from simple products like `python version X database backends` up to more diverse requirement setups like having different versions of different backends (for example different versions of the python scm bindings in anyvc) In order to express those i think 2 concepts are necessary. 1. Variations 2. Templates/blocks Variations express dimensions of variantion, for example python versions or database backends Blocks express a "base" for a testenv that will be enriched by the actual testenv to express the final testenv Blocks may extend/enrich upon other blocks (using the same mro algorithm python uses for inheritance) examples ========== Expressing the current python testenvs:: [part:pythons] variantions = python_version 24 25 26 27 30 31 32 22 python = python[${python_version}] #XXX: needs a sane way to evaluate python[24] = python2.4 python[25] = python2.5 python[26] = python2.6 python[27] = python2.7 python[30] = python3.0 python[31] = python3.1 python[32] = python3.1 python[33] = python3.3 [testenv:py${python_version}] use = pythons expressing multiple database backends:: [part:sqlite] depends = sqlite [part:mysql] depends = mysqldb [testenv:py$python_version-$backend] use = pythons ${backend} variantion = backend sqlite mysql postgres depends = ${previous} pytest commands = py.test --database ${backend} expressing more complex nested variation:: [part:mercurial] variation = version 2.0 2.1 2.2 depends = mercurial==${version} vcs = hg [part:subvertpy] version = latest depends = suvertpy==dev vcs = svn [testenv:py${python_version}-${backend}-{$version}] variations = backend subvertpy mercurial # override the variations of the pythons part python_version 26 27 33 use = pythons $backend depends = ${previous} pytest commands = py.test --vcs ${vcs} From holger at merlinux.eu Sat Jul 7 15:58:53 2012 From: holger at merlinux.eu (holger krekel) Date: Sat, 7 Jul 2012 13:58:53 +0000 Subject: [py-dev] (RFC) multi-dimensional/variant tox configuration (V1) Message-ID: <20120707135853.GL18336@merlinux.eu> Hi tox users, I'd like to find a good way to introduce multi-dimensional configuration to tox.ini files. I have written up a draft idea on how to do it and would appreciate feedback. I provide two examples of transformed tox.ini files. If you have suggestions (or example tox.ini which you would like to transform) i'd be grateful to hear about them. best & thanks, holger P.S.: Ronny Pfannschmidt just posted a different way to handle variants with tox, i'll check it up together with other feedback. V1 draft: multi-dimensional configuration with tox ------------------------------------------------------- Problems: - there is no way to define dependency or other variants in tox.ini files, instead you have explicitely spell out all combinations in separate testenvs. Examples: http://code.larlet.fr/django-rest-framework/src/eed0f39a7e45/tox.ini https://bitbucket.org/tabo/django-treebeard/src/93b579395a9c/tox.ini - tox always uses pip currently. So there is no check for your packages that installing with easy_install will work. Moreover, some packages, like greenlet on Win32, require easy_install if you have no suitable C-compiler on the machine. Tox cannot be used then currently. Goals: - allow to more easily define and run dependency/interpreter variants with testenvs - allow to run variants of installing via easy_install or pip. Generating and selecting variants ---------------------------------------------- Suppose you want to test your package against mypkg-1.3 and mypkg-1.4 framework versions and against python2.6,2.7,pypy-1.9 and 3.2 interpreters. Today you would have to create 2*4 = 8 ``[testenv*]`` sections to instruct tox to test against all of them. With tox-1.X you can use a new mechanism which is based on two ideas: * allow to specify partial testenv values in [testenv-VARIANT] sections * introduce "generator" expressions to the "envlist" setting to ease enumerating all variant combinations. Without much further introduction, here is an example ``tox.ini``:: envlist = py[26,27,32,py]-mypkg[13,14] [testenv] deps = nose commands = nosetests [testenv-mypkg13] +deps = mypkg<1.4 [testenv-django14] +deps = mypkg<1.5 If you don't want to run django-mypkg with pypy the envlist would look like this:: envlist = py[26,27,32]-mypkg[13,14] pypy-mypkg14 Generator expressions in the envlist setting ---------------------------------------------------------- Generator expressions in the ``envlist`` work like this: * ``[...]`` parts contain a comma-separated list of names. Each name will generate a new environment reference. * repeat the process until there are no more generator expressions Variant specification with [testenv-VARNAME] ---------------------------------------------- The ``[testenv-mypkg13]`` and ``[testenv-mypkg14]`` can define settings for the respective variant. Their specification can use a new ``+deps`` setting to allow for appending dependencies rather than replacing it. Showing all expanded sections ------------------------------- To help with understanding how the variants will produce section values, you can ask tox to show their expansion with a new option: $ tox -l [XXX output ommitted for now] Making sure your packages installs with easy_install ------------------------------------------------------ The new "installer" testenv setting allows to specify the tool for installation: [testenv] installer = easy_install If you want to have your package installed with both easy_install and pip, you can use variants again: [testenv-easy] installer = easy_install [testenv-pip] installer = pip [tox] envlist = py[26,27,32]-django[13,14]-[easy,pip] Note that tox comes with some predefined variants, namely: - [easy,pip] use easy_install or pip - [py24,py25,py26,py27,py31,py32,pypy,jy] use the respective pythonNN or PyPy or Jython interpreter You can use those in your "envlist" specification without the need to define them yourself. Transforming the examples: django-rest ------------------------------------------------ The original `tox.ini `_ file has 159 lines and a lot of repetition, the new one would have 26 lines and almost no repetition:: [tox] envlist = py[25,26,27]-django[12,13]-[example] [testenv] commands = python setup.py test deps= coverage==3.4 unittest-xml-reporting==1.2 Pyyaml==3.10 [testenv-django12] +deps= django==1.2.4 [testenv-django12] +deps= django==1.2.4 [testenv-example] deps = wsgiref==0.1.2 Pygments==1.4 httplib2==0.6.0 Markdown==2.0.3 commands = python examples/runtests.py Apart from the much more concise specification, it is now also easy to add further variants like testing installation with easy_install in addition to pip. Transforming the examples: django-treebeard ------------------------------------------------ Another `tox.ini `_ has 233 lines and runs tests against multiple Postgres and Mysql engines in the same testenv section. We can easily split this and reduce to 39 non-repetive lines:: [tox] envlist = py[24,25,26,27]-django[11,12,13]-[nodb,pg,mysql] docs [testenv:docs] changedir = docs deps = Sphinx Django commands = make clean make html [testenv] deps= coverage MySQL-python psycopg2 pysqlite [testenv-nodb] +deps=pysqlite commands = {envpython} runtests.py {posargs} [testenv-pg] +deps = psycopg2 commands = {envpython} runtests.py --DATABASE_ENGINE=postgresql_psycopg2 --DATABASE_USER=postgres {posargs} [testenv-mysql] +deps = MySQL-python commands = {envpython} runtests.py --DATABASE_ENGINE=mysql --DATABASE_USER=root {posargs} [testenv-django10] +deps = Django==1.0.4 From flub at devork.be Sat Jul 7 16:12:41 2012 From: flub at devork.be (Floris Bruynooghe) Date: Sat, 7 Jul 2012 16:12:41 +0200 Subject: [py-dev] Cached file tag Message-ID: Hi all, While I was fixing issue #168 I was wondering if it does not make sense to make the PYTEST_TAG variable (in _pytest.assertion.rewrite) available on the public API? Python itself has imp.get_tag(), so maybe pytest.get_tag() is a reasonable name? Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From benjamin at python.org Sat Jul 7 17:06:19 2012 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 7 Jul 2012 10:06:19 -0500 Subject: [py-dev] Cached file tag In-Reply-To: References: Message-ID: 2012/7/7 Floris Bruynooghe : > Hi all, > > While I was fixing issue #168 I was wondering if it does not make > sense to make the PYTEST_TAG variable (in _pytest.assertion.rewrite) > available on the public API? Python itself has imp.get_tag(), so > maybe pytest.get_tag() is a reasonable name? I assume you wanted this because you were using the tag in your test. test_rewrite.py is allowed to dig into the internals, so what you did was completely acceptable. I don't see what it would be needed for outside of that. It's very much an implementation detail. -- Regards, Benjamin From flub at devork.be Mon Jul 9 15:47:26 2012 From: flub at devork.be (Floris Bruynooghe) Date: Mon, 9 Jul 2012 14:47:26 +0100 Subject: [py-dev] Cached file tag In-Reply-To: References: Message-ID: On 7 July 2012 16:06, Benjamin Peterson wrote: > 2012/7/7 Floris Bruynooghe : >> While I was fixing issue #168 I was wondering if it does not make >> sense to make the PYTEST_TAG variable (in _pytest.assertion.rewrite) >> available on the public API? Python itself has imp.get_tag(), so >> maybe pytest.get_tag() is a reasonable name? > > I assume you wanted this because you were using the tag in your test. Not really, I was just drawing the parallel with imp.get_tag() which is also largely an implementation detail but still available. I have no problem with not providing it however. > test_rewrite.py is allowed to dig into the internals, so what you did > was completely acceptable. Sure, I have no problems with the test. (Btw, apologies for not following the file's style properly and not spotting where the internal imports where done, I shouldn't have written that is such a rush) Regards, Floris From issues-reply at bitbucket.org Mon Jul 9 21:25:36 2012 From: issues-reply at bitbucket.org (Ronny Pfannschmidt) Date: Mon, 09 Jul 2012 19:25:36 -0000 Subject: [py-dev] [hpk42/pytest-cache] option to run only new test if it applies (issue #3) Message-ID: <15a0cd1f817dee4852d79ba41041923f@bitbucket.org> --- you can reply above this line --- New issue 3: option to run only new test if it applies https://bitbucket.org/hpk42/pytest-cache/issue/3/option-to-run-only-new-test-if-it-applies Ronny Pfannschmidt: while creating a new test in a suite of already passing tests, it seems useful to have a way to run only new tests compared to the last run so a new failing test can be directly enter the loop, and i only get brothered with other tests again when it passes -- This is an issue notification from bitbucket.org. You are receiving this either because you are the owner of the issue, or you are following the issue. From benjamin at python.org Tue Jul 10 01:34:03 2012 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 9 Jul 2012 16:34:03 -0700 Subject: [py-dev] Cached file tag In-Reply-To: References: Message-ID: 2012/7/9 Floris Bruynooghe : > On 7 July 2012 16:06, Benjamin Peterson wrote: >> 2012/7/7 Floris Bruynooghe : >>> While I was fixing issue #168 I was wondering if it does not make >>> sense to make the PYTEST_TAG variable (in _pytest.assertion.rewrite) >>> available on the public API? Python itself has imp.get_tag(), so >>> maybe pytest.get_tag() is a reasonable name? >> >> I assume you wanted this because you were using the tag in your test. > > Not really, I was just drawing the parallel with imp.get_tag() which > is also largely an implementation detail but still available. I have > no problem with not providing it however. Let's wait until someone proposes a use for it. > >> test_rewrite.py is allowed to dig into the internals, so what you did >> was completely acceptable. > > Sure, I have no problems with the test. (Btw, apologies for not > following the file's style properly and not spotting where the > internal imports where done, I shouldn't have written that is such a > rush) No harm done. -- Regards, Benjamin From issues-reply at bitbucket.org Tue Jul 10 09:09:47 2012 From: issues-reply at bitbucket.org (Ronny Pfannschmidt) Date: Tue, 10 Jul 2012 07:09:47 -0000 Subject: [py-dev] [hpk42/pytest-cache] turning a failure into a skip by skip in setup will keep considering them failed (issue #4) Message-ID: <79d99f282af0dab3c7fc22246102678a@bitbucket.org> --- you can reply above this line --- New issue 4: turning a failure into a skip by skip in setup will keep considering them failed https://bitbucket.org/hpk42/pytest-cache/issue/4/turning-a-failure-into-a-skip-by-skip-in Ronny Pfannschmidt: the reason being a failure only being removed in the call passed phase (which does not happen in skip) -- This is an issue notification from bitbucket.org. You are receiving this either because you are the owner of the issue, or you are following the issue. From holger at merlinux.eu Tue Jul 10 12:09:00 2012 From: holger at merlinux.eu (holger krekel) Date: Tue, 10 Jul 2012 10:09:00 +0000 Subject: [py-dev] [TIP] (RFC) multi-dimensional/variant tox configuration (V1) In-Reply-To: <71E669C1-F260-499E-A8E0-67BDFA4508AC@sofa-rockers.org> References: <20120707135853.GL18336@merlinux.eu> <71E669C1-F260-499E-A8E0-67BDFA4508AC@sofa-rockers.org> Message-ID: <20120710100900.GU18336@merlinux.eu> Hi Stefan, On Tue, Jul 10, 2012 at 10:20 +0200, Stefan Scherfke wrote: > Hi Holger, > > I really like the idea. However, I found one bug and have one note: > > > Generating and selecting variants > > ---------------------------------------------- > > > > ? > > > > Without much further introduction, here is an example ``tox.ini``:: > > > > envlist = > > py[26,27,32,py]-mypkg[13,14] > > > > ? > > > > [testenv-mypkg13] > > +deps = mypkg<1.4 > > > > [testenv-django14] > > +deps = mypkg<1.5 > > I think it should be ?testenv-mypkg14? instead of ?testenv-django14?? right. > > If you don't want to run django-mypkg with pypy the envlist would look like > > this:: > > > > envlist = > > py[26,27,32]-mypkg[13,14] > > pypy-mypkg14 > > > > > > Generator expressions in the envlist setting > > ---------------------------------------------------------- > > > > Generator expressions in the ``envlist`` work like this: > > > > * ``[...]`` parts contain a comma-separated list of names. Each name > > will generate a new environment reference. > > * repeat the process until there are no more generator expressions > > I think you should make it more clear that: > > * you split the envlist entries by ?-? --> ['py[26,27,32]', 'mypkg[13,14]'] > * you then expand the generator expressions --> [['py26', 'py27', 'py32'], ['mypkg13', 'mypkg14']] > * and finally compute the cartesian product of that nested list. > * you can create a section for each item in the resulting list (i.e., ?py26? or ?mypkg13?) > * some of theses entries are predfined in tox (i.e. py26, pypy, ?) (you state this later, but it would be more helpful to remind the reader a bit earlier) It's a bit underspecified, i agree. The algorithm i had in mind works slightly different. Consider we have a list of environment names, some of which may contain [CSV]-generator expressions where the CSV part is a comma-separated list of variants. We then enter a loop as long as there are such expressions and then: - expand: for each CSV-expression in an environment name in the list produce an additional environment name for each value in the CSV - repeat: as long as there are CSV-expressions, continue the process Moreover, variants are defined by respective [testenv-VARIANT] sections. If neccessary, one can still override/special case a certain [testenv:VAR1-VAR2...] section by defining it. I believe it's all effectively very similar to what you describe except that there does not need to be special treatment of the "-" character. best, holger > > Cheers, > Stefan From issues-reply at bitbucket.org Tue Jul 10 13:58:14 2012 From: issues-reply at bitbucket.org (Ronny Pfannschmidt) Date: Tue, 10 Jul 2012 11:58:14 -0000 Subject: [py-dev] [hpk42/pytest-cache] deleted files leave items in the lastfail cache (issue #5) Message-ID: <171aa5ea61a27acfe82b78488bb903d6@bitbucket.org> --- you can reply above this line --- New issue 5: deleted files leave items in the lastfail cache https://bitbucket.org/hpk42/pytest-cache/issue/5/deleted-files-leave-items-in-the-lastfail Ronny Pfannschmidt: -- This is an issue notification from bitbucket.org. You are receiving this either because you are the owner of the issue, or you are following the issue. From parhamfh at kth.se Thu Jul 12 11:46:36 2012 From: parhamfh at kth.se (=?ISO-8859-1?Q?P=E4rham_Fazelzadeh_H?=) Date: Thu, 12 Jul 2012 11:46:36 +0200 Subject: [py-dev] KeyboardInterrupt during setup() and teardown() Message-ID: Hi all, I am using py.test to perform integration and functional testing of an application and had some issues with interrupts and was advised to submit my use case. Basically the problem is related to funcargs and how setup() and teardown() are affected by interrupts. The issue we are having is that some of our funcargs take longer to setup than what is maybe recommended. One of the funcargs instantiate and start the application that is to be tested and this can take some time; in the setup() you basically wait for the application to boot up to verify that it has started. If a KeyboardInterrupt is raised during setup() it will leave the application in a dirty state since no teardown will be run, i.e the application will be left running. Similarly this can happen during configuration stages in funcargs. I learned that this is default behaviour (and also reasonable), seeing as the idea is that funcargs should be small and be fast. Still, this is our use case so here you go! :) Regards, Parham -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Thu Jul 12 12:05:24 2012 From: holger at merlinux.eu (holger krekel) Date: Thu, 12 Jul 2012 10:05:24 +0000 Subject: [py-dev] KeyboardInterrupt during setup() and teardown() In-Reply-To: References: Message-ID: <20120712100524.GB18336@merlinux.eu> Hi P?rham, On Thu, Jul 12, 2012 at 11:46 +0200, P?rham Fazelzadeh H wrote: > Hi all, > > I am using py.test to perform integration and functional testing of an > application and had some issues with interrupts and was advised to submit > my use case. > > Basically the problem is related to funcargs and how setup() and teardown() > are affected by interrupts. The issue we are having is that some of our > funcargs take longer to setup than what is maybe recommended. One of the > funcargs instantiate and start the application that is to be tested and > this can take some time; in the setup() you basically wait for the > application to boot up to verify that it has started. If a > KeyboardInterrupt is raised during setup() it will leave the application in > a dirty state since no teardown will be run, i.e the application will be > left running. Similarly this can happen during configuration stages in > funcargs. > > I learned that this is default behaviour (and also reasonable), seeing as > the idea is that funcargs should be small and be fast. funcargs with a longer setup are fine. I am not sure i understand how you are missing teardowns. How do you perform teardown, with request.addfinalizer()? Can you provide a little examples that reproduces the problem? best, holger > Still, this is our use case so here you go! :) > > Regards, > Parham > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev From parhamfh at kth.se Thu Jul 12 14:12:55 2012 From: parhamfh at kth.se (=?ISO-8859-1?Q?P=E4rham_Fazelzadeh_H?=) Date: Thu, 12 Jul 2012 14:12:55 +0200 Subject: [py-dev] KeyboardInterrupt during setup() and teardown() In-Reply-To: <20120712100524.GB18336@merlinux.eu> References: <20120712100524.GB18336@merlinux.eu> Message-ID: Hi Holger, Here below is an example: def pytest_funcarg__app(request): def setup(): # Blocking call to start(), returns when application has finished booting up app.start() return app def teardown(val): app.stop() return request.cached_setup( setup=setup, teardown=teardown, scope='session', extrakey='app' ) Now, during setup(), when the process is waiting for the app to boot, if a keyboardinterrupt is raised (say by the user during from the test execution summary screen) then it will not call the teardown() of this funcarg 'app'. I assume this is because py.test assumes that the setup() of 'app' has not been finished and therefor it is not in a proper state for teardown() of 'app'. Regards, Parham On 12 July 2012 12:05, holger krekel wrote: > > Hi P?rham, > > On Thu, Jul 12, 2012 at 11:46 +0200, P?rham Fazelzadeh H wrote: > > Hi all, > > > > I am using py.test to perform integration and functional testing of an > > application and had some issues with interrupts and was advised to submit > > my use case. > > > > Basically the problem is related to funcargs and how setup() and teardown() > > are affected by interrupts. The issue we are having is that some of our > > funcargs take longer to setup than what is maybe recommended. One of the > > funcargs instantiate and start the application that is to be tested and > > this can take some time; in the setup() you basically wait for the > > application to boot up to verify that it has started. If a > > KeyboardInterrupt is raised during setup() it will leave the application in > > a dirty state since no teardown will be run, i.e the application will be > > left running. Similarly this can happen during configuration stages in > > funcargs. > > > > I learned that this is default behaviour (and also reasonable), seeing as > > the idea is that funcargs should be small and be fast. > > funcargs with a longer setup are fine. I am not sure i understand > how you are missing teardowns. How do you perform teardown, with > request.addfinalizer()? Can you provide a little examples that reproduces > the problem? > > best, > > holger > > > Still, this is our use case so here you go! :) > > > > Regards, > > Parham > > > _______________________________________________ > > py-dev mailing list > > py-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/py-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ronny.Pfannschmidt at gmx.de Thu Jul 12 15:26:45 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Thu, 12 Jul 2012 15:26:45 +0200 Subject: [py-dev] KeyboardInterrupt during setup() and teardown() In-Reply-To: References: <20120712100524.GB18336@merlinux.eu> Message-ID: <4FFED095.2020401@gmx.de> Hi P?rham, as i already explained in irc before, a cached setup only calls tear-down if it was successful, and if you want it to work property, you should split it up an example of doing that would be something like def pytest_funcarg__app(request): def setup() return create_app() # FAST def teardown(app): app.stop() app = request.cached_setup(setup, teardown, scope='session') app.start() #can wait return app -- Ronny On 07/12/2012 02:12 PM, P?rham Fazelzadeh H wrote: > Hi Holger, > > > Here below is an example: > > def pytest_funcarg__app(request): > > def setup(): > # Blocking call to start(), returns when application has > finished booting up > > app.start() > > return app > > > def teardown(val): > > app.stop() > > return request.cached_setup( > > setup=setup, > > teardown=teardown, > > scope='session', > > extrakey='app' > ) > > > Now, during setup(), when the process is waiting for the app to boot, if > a keyboardinterrupt is raised (say by the user during from the test > execution summary screen) then it will not call the teardown() of this > funcarg 'app'. I assume this is because py.test assumes that the setup() > of 'app' has not been finished and therefor it is not in a proper state > for teardown() of 'app'. > > Regards, > Parham > > On 12 July 2012 12:05, holger krekel > wrote: > > > > Hi P?rham, > > > > On Thu, Jul 12, 2012 at 11:46 +0200, P?rham Fazelzadeh H wrote: > > > Hi all, > > > > > > I am using py.test to perform integration and functional testing of an > > > application and had some issues with interrupts and was advised to > submit > > > my use case. > > > > > > Basically the problem is related to funcargs and how setup() and > teardown() > > > are affected by interrupts. The issue we are having is that some of our > > > funcargs take longer to setup than what is maybe recommended. One > of the > > > funcargs instantiate and start the application that is to be tested and > > > this can take some time; in the setup() you basically wait for the > > > application to boot up to verify that it has started. If a > > > KeyboardInterrupt is raised during setup() it will leave the > application in > > > a dirty state since no teardown will be run, i.e the application > will be > > > left running. Similarly this can happen during configuration stages in > > > funcargs. > > > > > > I learned that this is default behaviour (and also reasonable), > seeing as > > > the idea is that funcargs should be small and be fast. > > > > funcargs with a longer setup are fine. I am not sure i understand > > how you are missing teardowns. How do you perform teardown, with > > request.addfinalizer()? Can you provide a little examples that reproduces > > the problem? > > > > best, > > > > holger > > > > > Still, this is our use case so here you go! :) > > > > > > Regards, > > > Parham > > > > > _______________________________________________ > > > py-dev mailing list > > > py-dev at codespeak.net > > > http://codespeak.net/mailman/listinfo/py-dev > > > > > _______________________________________________ > py-dev mailing list > py-dev at codespeak.net > http://codespeak.net/mailman/listinfo/py-dev From holger at merlinux.eu Thu Jul 12 22:30:07 2012 From: holger at merlinux.eu (holger krekel) Date: Thu, 12 Jul 2012 20:30:07 +0000 Subject: [py-dev] KeyboardInterrupt during setup() and teardown() In-Reply-To: <4FFED095.2020401@gmx.de> References: <20120712100524.GB18336@merlinux.eu> <4FFED095.2020401@gmx.de> Message-ID: <20120712203007.GC18336@merlinux.eu> Hi P?rham, Ronny, On Thu, Jul 12, 2012 at 15:26 +0200, Ronny Pfannschmidt wrote: > Hi P?rham, > > as i already explained in irc before, > a cached setup only calls tear-down if it was successful, > > and if you want it to work property, you should split it up > > an example of doing that would be something like > > def pytest_funcarg__app(request): > def setup() > return create_app() # FAST > def teardown(app): > app.stop() > app = request.cached_setup(setup, teardown, scope='session') > app.start() #can wait > return app I don't think this is correct as it would start() the app in each test function that uses the "app" funcarg. It's indeed unusual that despite a failing setup you want to do some teardown. Maybe in this case controling it yourself is the best: def setup(): try: app.start() except Exception: # or KeyboardInterrupt etc. app.stop() raise hth, holger > -- Ronny > > On 07/12/2012 02:12 PM, P?rham Fazelzadeh H wrote: > >Hi Holger, > > > > > >Here below is an example: > > > >def pytest_funcarg__app(request): > > > > def setup(): > > # Blocking call to start(), returns when application has > >finished booting up > > > > app.start() > > > > return app > > > > > > def teardown(val): > > > > app.stop() > > > > return request.cached_setup( > > > > setup=setup, > > > > teardown=teardown, > > > > scope='session', > > > > extrakey='app' > > ) > > > > > >Now, during setup(), when the process is waiting for the app to boot, if > >a keyboardinterrupt is raised (say by the user during from the test > >execution summary screen) then it will not call the teardown() of this > >funcarg 'app'. I assume this is because py.test assumes that the setup() > >of 'app' has not been finished and therefor it is not in a proper state > >for teardown() of 'app'. > > > >Regards, > >Parham > > > >On 12 July 2012 12:05, holger krekel >> wrote: > > > > > > Hi P?rham, > > > > > > On Thu, Jul 12, 2012 at 11:46 +0200, P?rham Fazelzadeh H wrote: > > > > Hi all, > > > > > > > > I am using py.test to perform integration and functional testing of an > > > > application and had some issues with interrupts and was advised to > >submit > > > > my use case. > > > > > > > > Basically the problem is related to funcargs and how setup() and > >teardown() > > > > are affected by interrupts. The issue we are having is that some of our > > > > funcargs take longer to setup than what is maybe recommended. One > >of the > > > > funcargs instantiate and start the application that is to be tested and > > > > this can take some time; in the setup() you basically wait for the > > > > application to boot up to verify that it has started. If a > > > > KeyboardInterrupt is raised during setup() it will leave the > >application in > > > > a dirty state since no teardown will be run, i.e the application > >will be > > > > left running. Similarly this can happen during configuration stages in > > > > funcargs. > > > > > > > > I learned that this is default behaviour (and also reasonable), > >seeing as > > > > the idea is that funcargs should be small and be fast. > > > > > > funcargs with a longer setup are fine. I am not sure i understand > > > how you are missing teardowns. How do you perform teardown, with > > > request.addfinalizer()? Can you provide a little examples that reproduces > > > the problem? > > > > > > best, > > > > > > holger > > > > > > > Still, this is our use case so here you go! :) > > > > > > > > Regards, > > > > Parham > > > > > > > _______________________________________________ > > > > py-dev mailing list > > > > py-dev at codespeak.net > > > > http://codespeak.net/mailman/listinfo/py-dev > > > > > > > > >_______________________________________________ > >py-dev mailing list > >py-dev at codespeak.net > >http://codespeak.net/mailman/listinfo/py-dev > From Ronny.Pfannschmidt at gmx.de Thu Jul 12 23:23:32 2012 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Thu, 12 Jul 2012 23:23:32 +0200 Subject: [py-dev] KeyboardInterrupt during setup() and teardown() In-Reply-To: <20120712203007.GC18336@merlinux.eu> References: <20120712100524.GB18336@merlinux.eu> <4FFED095.2020401@gmx.de> <20120712203007.GC18336@merlinux.eu> Message-ID: <4FFF4054.1050901@gmx.de> Hi all, my bad, that start call should have been named ensure_started On 07/12/2012 10:30 PM, holger krekel wrote: > Hi P?rham, Ronny, > > On Thu, Jul 12, 2012 at 15:26 +0200, Ronny Pfannschmidt wrote: >> Hi P?rham, >> >> as i already explained in irc before, >> a cached setup only calls tear-down if it was successful, >> >> and if you want it to work property, you should split it up >> >> an example of doing that would be something like >> >> def pytest_funcarg__app(request): >> def setup() >> return create_app() # FAST >> def teardown(app): >> app.stop() >> app = request.cached_setup(setup, teardown, scope='session') >> app.start() #can wait >> return app > > I don't think this is correct as it would start() the app > in each test function that uses the "app" funcarg. > > It's indeed unusual that despite a failing setup you > want to do some teardown. Maybe in this case controling > it yourself is the best: > > def setup(): > try: > app.start() > except Exception: # or KeyboardInterrupt etc. > app.stop() > raise > > hth, > holger > > http://codespeak.net/mailman/listinfo/py-dev From holger at merlinux.eu Fri Jul 20 14:25:32 2012 From: holger at merlinux.eu (holger krekel) Date: Fri, 20 Jul 2012 12:25:32 +0000 Subject: [py-dev] direct funcarg scoping/parametrization implementation (and resource-v3 draft) Message-ID: <20120720122532.GJ26019@merlinux.eu> Hi Floris, Ronny, Carl, all, i've managed to do a first round of implementation of the recently discussed resources API. For an example on what is now possible see: http://pytest.org/dev/example/newexamples.html As far as i see the new features did not break backward-compatibility with existing plugin or test code. I've also written a V3-work-in-progress of the resources document and marked the status of features in each section: http://pytest.org/dev/resources.html If you'd like to try things out you can either grab it via:: pip install -i http://pypi.testrun.org -U pytest which should give a "py.test --version" >= 2.3.0.dev3 or you can go to the bitbucket repository. For the next step, i am considering if to try to extend setup_X methods of if to introduce a new @pytest.mark.setup marker, see the resources.html doc. I lean towards the letter and believe that its introduction would make a lot of pytest_runtest_setup() and pytest_configure() unneccessary. grateful for any feedback or comments, holger From carl at oddbird.net Fri Jul 20 20:08:45 2012 From: carl at oddbird.net (Carl Meyer) Date: Fri, 20 Jul 2012 12:08:45 -0600 Subject: [py-dev] direct funcarg scoping/parametrization implementation (and resource-v3 draft) In-Reply-To: <20120720122532.GJ26019@merlinux.eu> References: <20120720122532.GJ26019@merlinux.eu> Message-ID: <50099EAD.1010805@oddbird.net> Hi Holger, I love the pytest.mark.funcarg decorator. I think pytest.mark.setup is likely a good idea, too, but there are some questions I'm not clear on: 1. How do I handle teardown for these setup functions? I would expect they'd take a request and I'd do request.addfinalizer(...), but in some of your examples they don't seem to take request, and in the one where it does, it says "In addition to normal funcargs you can also receive the ?request? funcarg which represents a takes on each of the values in the params=[1,2,3] decorator argument" - which I'm having trouble parsing, and it isn't clear to me this request object would have addfinalizer(). 2. It's not entirely clear how the two types of scope Floris referred to earlier (scope based on location of the decorated function, and the scope keyword passed to the decorator) interact with each other. I presume that if I have a setup-decorated function in a conftest.py, it only applies to that directory and subdirectories. If it's located in a module, I guess it only applies to tests in that module? What if it's located in a module and I give it scope="session" - what does that mean? Would that be functionally equivalent to scope="module" in that case, since it still only applies to that module? Similarly, if I decorate a method of a class with the setup marker, does it only apply to test methods on that class? 3. Is there a "class" value for the scope kwarg, in addition to "session", "module", and "function"? It would be nice to see a full list of the accepted values for that kwarg. That's all that comes to mind at the moment! Thanks for all your work on this. Carl From holger at merlinux.eu Sat Jul 21 01:25:52 2012 From: holger at merlinux.eu (holger krekel) Date: Fri, 20 Jul 2012 23:25:52 +0000 Subject: [py-dev] direct funcarg scoping/parametrization implementation (and resource-v3 draft) In-Reply-To: <50099EAD.1010805@oddbird.net> References: <20120720122532.GJ26019@merlinux.eu> <50099EAD.1010805@oddbird.net> Message-ID: <20120720232552.GN26019@merlinux.eu> Hi Carl, On Fri, Jul 20, 2012 at 12:08 -0600, Carl Meyer wrote: > Hi Holger, > > I love the pytest.mark.funcarg decorator. Good to hear! I also like it :) And i think it makes sense to just extend the funcargs system rather than to invent a parallel "resources" one. > I think pytest.mark.setup is likely a good idea, too, but there are some > questions I'm not clear on: > > 1. How do I handle teardown for these setup functions? I would expect > they'd take a request and I'd do request.addfinalizer(...), but in some > of your examples they don't seem to take request, and in the one where > it does, it says "In addition to normal funcargs you can also receive > the ?request? funcarg which represents a takes on each of the values in > the params=[1,2,3] decorator argument" - which I'm having trouble > parsing, and it isn't clear to me this request object would have > addfinalizer(). Let me fix the paragraph, it should read something like: This would execute the ``modes`` function once for each parameter which will be put at ``request.param``. This request object offers the ``addfinalizer(func)`` helper which allows to register a function which will be executed when test functions within the specified scope finished execution. The ``request`` is a funcarg and thus setup functions can choose to receive it or not by stating it in their signature. It will always be available. Depending on the scope, ``request.node`` will be the corresponding node i guess. > 2. It's not entirely clear how the two types of scope Floris referred to > earlier (scope based on location of the decorated function, and the > scope keyword passed to the decorator) interact with each other. I > presume that if I have a setup-decorated function in a conftest.py, it > only applies to that directory and subdirectories. right. > If it's located in a > module, I guess it only applies to tests in that module? Yes, this has been the rule of pytest_funcarg__ and pytest_generate_tests definitions and thus it makes sense to use the same logic i think. > What if it's > located in a module and I give it scope="session" - what does that mean? > Would that be functionally equivalent to scope="module" in that case, > since it still only applies to that module? Yes, exactly. > Similarly, if I decorate a > method of a class with the setup marker, does it only apply to test > methods on that class? Yes. In this case, session, module and class would all have the same meaning or at least almost the same. Currently the effectively tighter scope is not detected so a session-scoped finalizer will be executed at the end of a session and not at the end of the class. > 3. Is there a "class" value for the scope kwarg, in addition to > "session", "module", and "function"? It would be nice to see a full list > of the accepted values for that kwarg. so far: session, module, class, function i am thinking about directory - it's actually slightly tricky to implement so i may postpone until someone really wants it :) However, what i am thinking about is allowing to specify a function as scope. In that case, something like function(config) would be called so that you could define scope according to command line options. This is useful when you want to have slow but better isolated per-function scopes for resources in CI runs. > That's all that comes to mind at the moment! Thanks for all your work on > this. Thanks for all your feedback, it's really helpful. My next goal is to implement the setup-functions and do some more examples. I believe some pretty cool things can be done with it, one example is doing per-function transactions on a session-scoped database:: # content of conftest.py import pytest @pytest.mark.funcarg(scope="session", ...) def db(request): ... @pytest.mark.setup(scope="function") def dbtransact(request, db): if should_transact(request.node.obj ...): db.begin() request.addfinalizer(db.end) Here, test functions themselves do not need to require "db" because th setup function requires it. This way you can manage global resources without explicitely caring from test functions. best, holger P.S: sometimes i wonder if a web framework couldn't use similar ideas as funcargs/scoping/declarations/etc. for implementing interactions between components ... But don't worry, i am not going to implement my own any time soon :) From carl at oddbird.net Sat Jul 21 01:40:24 2012 From: carl at oddbird.net (Carl Meyer) Date: Fri, 20 Jul 2012 17:40:24 -0600 Subject: [py-dev] direct funcarg scoping/parametrization implementation (and resource-v3 draft) In-Reply-To: <20120720232552.GN26019@merlinux.eu> References: <20120720122532.GJ26019@merlinux.eu> <50099EAD.1010805@oddbird.net> <20120720232552.GN26019@merlinux.eu> Message-ID: <5009EC68.70702@oddbird.net> On 07/20/2012 05:25 PM, holger krekel wrote: >> 1. How do I handle teardown for these setup functions? I would expect >> they'd take a request and I'd do request.addfinalizer(...), but in some >> of your examples they don't seem to take request, and in the one where >> it does, it says "In addition to normal funcargs you can also receive >> the ?request? funcarg which represents a takes on each of the values in >> the params=[1,2,3] decorator argument" - which I'm having trouble >> parsing, and it isn't clear to me this request object would have >> addfinalizer(). > > Let me fix the paragraph, it should read something like: > > This would execute the ``modes`` function once for each parameter > which will be put at ``request.param``. This request object offers > the ``addfinalizer(func)`` helper which allows to register a function > which will be executed when test functions within the specified scope > finished execution. > > The ``request`` is a funcarg and thus setup functions can choose to > receive it or not by stating it in their signature. It will always > be available. Depending on the scope, ``request.node`` will be the > corresponding node i guess. Yep, that would definitely clarify it. Carl From lpbrac at dolby.com Thu Jul 26 21:39:15 2012 From: lpbrac at dolby.com (Brack, Laurent P.) Date: Thu, 26 Jul 2012 12:39:15 -0700 Subject: [py-dev] [PY-DEV] Accessing "Item" property from a test case Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE4797D2CF@sfo-exch-01.dolby.net> I was wondering if there was any way for a test case to retrieve a property containing arbitrary data that could be set by a hook? An example could be a logger that has been opened specially for that test or conversely, the test could "attach" data which can be processed by a hook (example: test notes containing measurements, etc.) One could argue that the logger could be created by a factory but in this case, the test is correlated to a specific test case ID on our test management system (testlink) and this information is not available from a factory. In one of our plugin, I am currently attaching "metadata" to "items" to convey information between hooks and I wished there was a way for the test itself to access this. Maybe there is, I just don't know about it. Thanks in advance /Laurent -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Fri Jul 27 06:41:01 2012 From: holger at merlinux.eu (holger krekel) Date: Fri, 27 Jul 2012 04:41:01 +0000 Subject: [py-dev] [PY-DEV] Accessing "Item" property from a test case In-Reply-To: <05BB196AB3DA6C4BBE11AB6C957581FE4797D2CF@sfo-exch-01.dolby.net> References: <05BB196AB3DA6C4BBE11AB6C957581FE4797D2CF@sfo-exch-01.dolby.net> Message-ID: <20120727044101.GW26019@merlinux.eu> Hi Laurent, On Thu, Jul 26, 2012 at 12:39 -0700, Brack, Laurent P. wrote: > I was wondering if there was any way for a test case to retrieve a > property containing arbitrary data that could be set by a hook? Funcargs look like the natural place for this, http://pytest.org/latest/funcargs.html > An example could be a logger that has been opened specially for that > test or conversely, the test could "attach" data which can be processed > by a hook (example: test notes containing measurements, etc.) One could > argue that the logger could be created by a factory but in this case, > the test is correlated to a specific test case ID on our test management > system (testlink) and this information is not available from a factory. Almost all information is available within funcarg factories and can thus be made available to the test through injection. For example, if you have a decorator with a testcase-id then the factory can read it and pass it along to the test etc. > In one of our plugin, I am currently attaching "metadata" to "items" to > convey information between hooks and I wished there was a way for the > test itself to access this. Maybe there is, I just don't know about it. I need some simple example to understand why funcargs would not work if they won't :) best, holger