Mailman 3 July 2012 - pytest-dev

[py-dev] RFC: V2 of the new resource setup/parametrization facilities
by holger krekel Aug. 7, 2012

Aug. 7, 2012

Hi all, particularly Floris and Carl, i have finally arrived at the V2 resource-API draft based on the very valuable feedback you gave to the first version. The document implements a largely changed approach, see the "Changes from V1 to V2" at the beginning, and focuses on usage-level documentation instead of internal details. I have also uploaded this doc as HTML which makes it a bit more colorful to read, and also contains some cross-references: http://pytest.org/dev/resources.html Please find the the source txt-file also attached for your inline-commenting usage. Before i target the actual (substantial) refactoring, i'd actually be very grateful for some more of your time and comments on this new version. I believe that the new resource parametrization facilities are a major step forward - they should allow test writers to much more seldomly having to resort to pytest_* hooks, and make access and working with parametrized resources straight forward, irrespective of previous xUnit/pytest background. Plugin writers, of course, may still use the hooks for good value. best & thanks, holger

3 11

[py-dev] [PY-DEV] Accessing "Item" property from a test case
by Brack, Laurent P. July 27, 2012

July 27, 2012

I was wondering if there was any way for a test case to retrieve a property containing arbitrary data that could be set by a hook? An example could be a logger that has been opened specially for that test or conversely, the test could "attach" data which can be processed by a hook (example: test notes containing measurements, etc.) One could argue that the logger could be created by a factory but in this case, the test is correlated to a specific test case ID on our test management system (testlink) and this information is not available from a factory. In one of our plugin, I am currently attaching "metadata" to "items" to convey information between hooks and I wished there was a way for the test itself to access this. Maybe there is, I just don't know about it. Thanks in advance /Laurent

2 1

[py-dev] direct funcarg scoping/parametrization implementation (and resource-v3 draft)
by holger krekel July 20, 2012

July 20, 2012

Hi Floris, Ronny, Carl, all, i've managed to do a first round of implementation of the recently discussed resources API. For an example on what is now possible see: http://pytest.org/dev/example/newexamples.html As far as i see the new features did not break backward-compatibility with existing plugin or test code. I've also written a V3-work-in-progress of the resources document and marked the status of features in each section: http://pytest.org/dev/resources.html If you'd like to try things out you can either grab it via:: pip install -i http://pypi.testrun.org -U pytest which should give a "py.test --version" >= 2.3.0.dev3 or you can go to the bitbucket repository. For the next step, i am considering if to try to extend setup_X methods of if to introduce a new @pytest.mark.setup marker, see the resources.html doc. I lean towards the letter and believe that its introduction would make a lot of pytest_runtest_setup() and pytest_configure() unneccessary. grateful for any feedback or comments, holger

2 3

[py-dev] KeyboardInterrupt during setup() and teardown()
by Pärham Fazelzadeh H July 12, 2012

July 12, 2012

Hi all, I am using py.test to perform integration and functional testing of an application and had some issues with interrupts and was advised to submit my use case. Basically the problem is related to funcargs and how setup() and teardown() are affected by interrupts. The issue we are having is that some of our funcargs take longer to setup than what is maybe recommended. One of the funcargs instantiate and start the application that is to be tested and this can take some time; in the setup() you basically wait for the application to boot up to verify that it has started. If a KeyboardInterrupt is raised during setup() it will leave the application in a dirty state since no teardown will be run, i.e the application will be left running. Similarly this can happen during configuration stages in funcargs. I learned that this is default behaviour (and also reasonable), seeing as the idea is that funcargs should be small and be fast. Still, this is our use case so here you go! :) Regards, Parham

3 5

[py-dev] [hpk42/pytest-cache] deleted files leave items in the lastfail cache (issue #5)
by Ronny Pfannschmidt July 10, 2012

July 10, 2012

--- you can reply above this line --- New issue 5: deleted files leave items in the lastfail cache https://bitbucket.org/hpk42/pytest-cache/issue/5/deleted-files-leave-items-… Ronny Pfannschmidt: -- This is an issue notification from bitbucket.org. You are receiving this either because you are the owner of the issue, or you are following the issue.

1 0

Re: [py-dev] [TIP] (RFC) multi-dimensional/variant tox configuration (V1)
by holger krekel July 10, 2012

July 10, 2012

Hi Stefan, On Tue, Jul 10, 2012 at 10:20 +0200, Stefan Scherfke wrote: > Hi Holger, > > I really like the idea. However, I found one bug and have one note: > > > Generating and selecting variants > > ---------------------------------------------- > > > > … > > > > Without much further introduction, here is an example ``tox.ini``:: > > > > envlist = > > py[26,27,32,py]-mypkg[13,14] > > > > … > > > > [testenv-mypkg13] > > +deps = mypkg<1.4 > > > > [testenv-django14] > > +deps = mypkg<1.5 > > I think it should be “testenv-mypkg14” instead of “testenv-django14”? right. > > If you don't want to run django-mypkg with pypy the envlist would look like > > this:: > > > > envlist = > > py[26,27,32]-mypkg[13,14] > > pypy-mypkg14 > > > > > > Generator expressions in the envlist setting > > ---------------------------------------------------------- > > > > Generator expressions in the ``envlist`` work like this: > > > > * ``[...]`` parts contain a comma-separated list of names. Each name > > will generate a new environment reference. > > * repeat the process until there are no more generator expressions > > I think you should make it more clear that: > > * you split the envlist entries by “-” --> ['py[26,27,32]', 'mypkg[13,14]'] > * you then expand the generator expressions --> [['py26', 'py27', 'py32'], ['mypkg13', 'mypkg14']] > * and finally compute the cartesian product of that nested list. > * you can create a section for each item in the resulting list (i.e., “py26” or “mypkg13”) > * some of theses entries are predfined in tox (i.e. py26, pypy, …) (you state this later, but it would be more helpful to remind the reader a bit earlier) It's a bit underspecified, i agree. The algorithm i had in mind works slightly different. Consider we have a list of environment names, some of which may contain [CSV]-generator expressions where the CSV part is a comma-separated list of variants. We then enter a loop as long as there are such expressions and then: - expand: for each CSV-expression in an environment name in the list produce an additional environment name for each value in the CSV - repeat: as long as there are CSV-expressions, continue the process Moreover, variants are defined by respective [testenv-VARIANT] sections. If neccessary, one can still override/special case a certain [testenv:VAR1-VAR2...] section by defining it. I believe it's all effectively very similar to what you describe except that there does not need to be special treatment of the "-" character. best, holger > > Cheers, > Stefan

1 0

[py-dev] [hpk42/pytest-cache] turning a failure into a skip by skip in setup will keep considering them failed (issue #4)
by Ronny Pfannschmidt July 10, 2012

July 10, 2012

--- you can reply above this line --- New issue 4: turning a failure into a skip by skip in setup will keep considering them failed https://bitbucket.org/hpk42/pytest-cache/issue/4/turning-a-failure-into-a-s… Ronny Pfannschmidt: the reason being a failure only being removed in the call passed phase (which does not happen in skip) -- This is an issue notification from bitbucket.org. You are receiving this either because you are the owner of the issue, or you are following the issue.

1 0

[py-dev] Cached file tag
by Floris Bruynooghe July 9, 2012

July 9, 2012

Hi all, While I was fixing issue #168 I was wondering if it does not make sense to make the PYTEST_TAG variable (in _pytest.assertion.rewrite) available on the public API? Python itself has imp.get_tag(), so maybe pytest.get_tag() is a reasonable name? Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

2 3

[py-dev] [hpk42/pytest-cache] option to run only new test if it applies (issue #3)
by Ronny Pfannschmidt July 9, 2012

July 9, 2012

--- you can reply above this line --- New issue 3: option to run only new test if it applies https://bitbucket.org/hpk42/pytest-cache/issue/3/option-to-run-only-new-tes… Ronny Pfannschmidt: while creating a new test in a suite of already passing tests, it seems useful to have a way to run only new tests compared to the last run so a new failing test can be directly enter the loop, and i only get brothered with other tests again when it passes -- This is an issue notification from bitbucket.org. You are receiving this either because you are the owner of the issue, or you are following the issue.

1 0

[py-dev] (RFC) multi-dimensional/variant tox configuration (V1)
by holger krekel July 7, 2012

July 7, 2012

Hi tox users, I'd like to find a good way to introduce multi-dimensional configuration to tox.ini files. I have written up a draft idea on how to do it and would appreciate feedback. I provide two examples of transformed tox.ini files. If you have suggestions (or example tox.ini which you would like to transform) i'd be grateful to hear about them. best & thanks, holger P.S.: Ronny Pfannschmidt just posted a different way to handle variants with tox, i'll check it up together with other feedback. V1 draft: multi-dimensional configuration with tox ------------------------------------------------------- Problems: - there is no way to define dependency or other variants in tox.ini files, instead you have explicitely spell out all combinations in separate testenvs. Examples: http://code.larlet.fr/django-rest-framework/src/eed0f39a7e45/tox.ini https://bitbucket.org/tabo/django-treebeard/src/93b579395a9c/tox.ini - tox always uses pip currently. So there is no check for your packages that installing with easy_install will work. Moreover, some packages, like greenlet on Win32, require easy_install if you have no suitable C-compiler on the machine. Tox cannot be used then currently. Goals: - allow to more easily define and run dependency/interpreter variants with testenvs - allow to run variants of installing via easy_install or pip. Generating and selecting variants ---------------------------------------------- Suppose you want to test your package against mypkg-1.3 and mypkg-1.4 framework versions and against python2.6,2.7,pypy-1.9 and 3.2 interpreters. Today you would have to create 2*4 = 8 ``[testenv*]`` sections to instruct tox to test against all of them. With tox-1.X you can use a new mechanism which is based on two ideas: * allow to specify partial testenv values in [testenv-VARIANT] sections * introduce "generator" expressions to the "envlist" setting to ease enumerating all variant combinations. Without much further introduction, here is an example ``tox.ini``:: envlist = py[26,27,32,py]-mypkg[13,14] [testenv] deps = nose commands = nosetests [testenv-mypkg13] +deps = mypkg<1.4 [testenv-django14] +deps = mypkg<1.5 If you don't want to run django-mypkg with pypy the envlist would look like this:: envlist = py[26,27,32]-mypkg[13,14] pypy-mypkg14 Generator expressions in the envlist setting ---------------------------------------------------------- Generator expressions in the ``envlist`` work like this: * ``[...]`` parts contain a comma-separated list of names. Each name will generate a new environment reference. * repeat the process until there are no more generator expressions Variant specification with [testenv-VARNAME] ---------------------------------------------- The ``[testenv-mypkg13]`` and ``[testenv-mypkg14]`` can define settings for the respective variant. Their specification can use a new ``+deps`` setting to allow for appending dependencies rather than replacing it. Showing all expanded sections ------------------------------- To help with understanding how the variants will produce section values, you can ask tox to show their expansion with a new option: $ tox -l [XXX output ommitted for now] Making sure your packages installs with easy_install ------------------------------------------------------ The new "installer" testenv setting allows to specify the tool for installation: [testenv] installer = easy_install If you want to have your package installed with both easy_install and pip, you can use variants again: [testenv-easy] installer = easy_install [testenv-pip] installer = pip [tox] envlist = py[26,27,32]-django[13,14]-[easy,pip] Note that tox comes with some predefined variants, namely: - [easy,pip] use easy_install or pip - [py24,py25,py26,py27,py31,py32,pypy,jy] use the respective pythonNN or PyPy or Jython interpreter You can use those in your "envlist" specification without the need to define them yourself. Transforming the examples: django-rest ------------------------------------------------ The original `tox.ini <http://code.larlet.fr/django-rest-framework/src/eed0f39a7e45/tox.ini>`_ file has 159 lines and a lot of repetition, the new one would have 26 lines and almost no repetition:: [tox] envlist = py[25,26,27]-django[12,13]-[example] [testenv] commands = python setup.py test deps= coverage==3.4 unittest-xml-reporting==1.2 Pyyaml==3.10 [testenv-django12] +deps= django==1.2.4 [testenv-django12] +deps= django==1.2.4 [testenv-example] deps = wsgiref==0.1.2 Pygments==1.4 httplib2==0.6.0 Markdown==2.0.3 commands = python examples/runtests.py Apart from the much more concise specification, it is now also easy to add further variants like testing installation with easy_install in addition to pip. Transforming the examples: django-treebeard ------------------------------------------------ Another `tox.ini <https://bitbucket.org/tabo/django-treebeard/raw/93b579395a9c/tox.ini>`_ has 233 lines and runs tests against multiple Postgres and Mysql engines in the same testenv section. We can easily split this and reduce to 39 non-repetive lines:: [tox] envlist = py[24,25,26,27]-django[11,12,13]-[nodb,pg,mysql] docs [testenv:docs] changedir = docs deps = Sphinx Django commands = make clean make html [testenv] deps= coverage MySQL-python psycopg2 pysqlite [testenv-nodb] +deps=pysqlite commands = {envpython} runtests.py {posargs} [testenv-pg] +deps = psycopg2 commands = {envpython} runtests.py --DATABASE_ENGINE=postgresql_psycopg2 --DATABASE_USER=postgres {posargs} [testenv-mysql] +deps = MySQL-python commands = {envpython} runtests.py --DATABASE_ENGINE=mysql --DATABASE_USER=root {posargs} [testenv-django10] +deps = Django==1.0.4

1 0