Mailman 3 June 2012 - pytest-dev

[py-dev] RFC: V2 of the new resource setup/parametrization facilities
by holger krekel Aug. 7, 2012

Aug. 7, 2012

Hi all, particularly Floris and Carl, i have finally arrived at the V2 resource-API draft based on the very valuable feedback you gave to the first version. The document implements a largely changed approach, see the "Changes from V1 to V2" at the beginning, and focuses on usage-level documentation instead of internal details. I have also uploaded this doc as HTML which makes it a bit more colorful to read, and also contains some cross-references: http://pytest.org/dev/resources.html Please find the the source txt-file also attached for your inline-commenting usage. Before i target the actual (substantial) refactoring, i'd actually be very grateful for some more of your time and comments on this new version. I believe that the new resource parametrization facilities are a major step forward - they should allow test writers to much more seldomly having to resort to pytest_* hooks, and make access and working with parametrized resources straight forward, irrespective of previous xUnit/pytest background. Plugin writers, of course, may still use the hooks for good value. best & thanks, holger

3 11

[py-dev] RFC: draft new resource management API (v1)
by holger krekel June 29, 2012

June 29, 2012

Hi all, based on on initial discussions with Ronny and Floris i have now written a usage-level document for a new test resource management API. It aims to better support plugin and test writers in managing cross-test-suite resources such as databases, temporary directories, etc. It generalizes the existing funcarg factory mechanism - currently some knowledge of such pytest usages is required to understand the document. Also, it does not fully spell out all details yet - i hope it nevertheless transports the main ideas. Happy about any feedback, holger V1: Creating and working with test resources ============================================== pytest-2.3 provides generalized resource management allowing to flexibly manage caching and parametrization across your test suite. This is draft documentation, pending refinements and changes according to feedback and to implementation or backward compatibility issues (the new mechanism is supposed to allow fully backward compatible operations for uses of the "funcarg" mechanism). the new global pytest_runtest_init hook ------------------------------------------------------ Prior to 2.3, pytest offered a pytest_configure and a pytest_sessionstart hook which was used often to setup global resources. This suffers from several problems. First of all, in distributed testing the master would also setup test resources that are never needed because it only co-ordinates the test run activities of the slave processes. Secondly, in large test suites resources are setup that might not be needed for the concrete test run. The first issue is solved through the introduction of a specific hook:: def pytest_runtest_init(session): # called ahead of pytest_runtestloop() test execution This hook will only be called in processes that actually run tests. The second issue is solved through a new register/getresource API which will only ever setup resources if they are needed. See the following examples and sections on how this works. managing a global database resource --------------------------------------------------------------- If you have one database object which you want to use in tests you can write the following into a conftest.py file:: class Database: def __init__(self): print ("database instance created") def destroy(self): print ("database instance destroyed") def factory_db(name, node): db = Database() node.addfinalizer(db.destroy) return db def pytest_runtest_init(session): session.register_resource("db", factory_db, atnode=session) You can then access the constructed resource in a test like this:: def test_something(db): ... The "db" function argument will lead to a lookup of the respective factory value and be passed to the function body. According to the registration, the db object will be instantiated on a per-session basis and thus reused across all test functions that require it. instantiating a database resource per-module --------------------------------------------------------------- If you want one database instance per test module you can restrict caching by modifying the "atnode" parameter of the registration call above:: def pytest_runtest_init(session): session.register_resource("db", factory_db, atnode=pytest.Module) Neither the tests nor the factory function will need to change. This also means that you can decide the scoping of resources at runtime - e.g. based on a command line option: for developer settings you might want per-session and for Continous Integration runs you might prefer per-module or even per-function scope like this:: def pytest_runtest_init(session): session.register_resource_factory("db", factory_db, atnode=pytest.Function) parametrized resources ---------------------------------- If you want to rerun tests with different resource values you can specify a list of factories instead of just one:: def pytest_runtest_init(session): session.register_factory("db", [factory1, factory2], atnode=session) In this case all tests that depend on the "db" resource will be run twice using the respective values obtained from the two factory functions. Using a resource from another resource factory ---------------------------------------------- You can use the database resource from a another resource factory through the ``node.getresource()`` method. Let's add a resource factory for a "db_users" table at module-level, extending the previous db-example:: def pytest_runtest_init(session): ... session.register_factory("db_users", createusers, atnode=module) def createusers(name, node): db = node.getresource("db") table = db.create_table("users", ...) node.addfinalizer(lambda: db.destroy_table("users") def test_user_creation(db_users): ... The create-users will be called for each module. After the tests in that module finish execution, the table will be destroyed according to registered finalizer. Note that calling getresource() for a resource which has a tighter scope will raise a LookupError because the is not available at a more general scope. Concretely, if you table is defined as a per-session resource and the database object as a per-module one, the table creation cannot work on a per-session basis. Setting resources as class attributes ------------------------------------------- If you want to make an attribute available on a test class, you can use the resource_attr marker:: @pytest.mark.resource_attr("db") class TestClass: def test_something(self): #use self.db Note that this way of using resources can be used on unittest.TestCase instances as well (function arguments can not be added due to unittest limitations). How the funcarg mechanism is implemented (internal notes) ------------------------------------------------------------- Prior to pytest-2.3/4, pytest advertised the "funcarg" mechanism which provided a subset functionality to the generalized resource management. In fact, the previous mechanism is implemented in terms of the new API and should continue to work unmodified. It basically automates the registration of factories through automatic discovery of ``pytest_funcarg_NAME`` function on plugins, Python modules and classes. As an example let's consider the Module.setup() method:: class Module(PyCollector): def setup(self): for name, func in self.obj.__dict__.items(): if name.startswith("pytest_funcarg__"): resourcename = name[len("pytest_funcarg__"):] self._register_factory(resourcename, RequestAdapter(self, name, func)) The request adapater takes care to provide the pre-2.3 API for funcarg factories, providing request.cached_setup/addfinalizer/getfuncargvalue methods.

2 8

Re: [py-dev] RFC: draft new resource management API (v1)
by holger krekel June 27, 2012

June 27, 2012

On Wed, Jun 27, 2012 at 08:43 -0600, Carl Meyer wrote: > I like it! In particular the parametrization support by passing a list > is a quite intuitive extension of the API. > > "atnode" seems like an opaque arg name - what's wrong with "scope"? The > latter name seems more intuitive to me. Would this arg have a default value? "scope" makes sense - it's just that in the current API scope is a "class", "module", ... string. Existing users might easily get a bit of type clash - especially if you have a mixed funcarg/resource scenario. Maybe "scopenode"? The default scopenode would be the one on which you are calling "register_factory". So in the first documented example call: session.register_factory("db", createdb, scopenode=session) the "scopenode" call would actually be superfluous. (Sidenote: the session object is also a node - the root node from which all collection and item nodes are descendants. Each node has a ".session" reference back to this root node). > In the long run, if funcarg-style is considered a useful shortcut and > will not be deprecated, it would be nice if there were a bit more naming > and API consistency between funcarg-style and new-style resource > handling -- it would make them feel more aspects of one system rather > than two different systems. I think this would really just require > switching from pytest_funcarg__foo to pytest_resource__foo, renaming > cached_setup to register_factory (and having it use the same API), and > renaming getfuncargvalue to getresource. Of course I don't know whether > this consistency is really worth the backwards-compatibility/deprecation > hassles. * getresource/getfuncargvalue: makes sense to me to go for advertising and documenting getresource() instead of getfuncargvalue() and keeping the latter as an alias with or without deprecation. * addfinalizer would remain unmodified - it's just that the "request" object passed to funcarg-factories adds finalizers with test function invocation scope, whereas node.addfinalizer() does it for the respective node scope (so e.g. called from a Class node it would register a per-class finalizer) * cached_setup: i hope that we do not need to offer this method anymore other than for compatibility. It's internal caching-key is not easy to explain and more than once users have stumbled about understanding it. cached_setup is required as long as pytest_funcarg__ factories are called _each_ time a resource is requested. (By contrast the new getresource() only triggers a factory call once for the registered scope - thus the factory implementation itself does not need to care for caching). Note that register_factory is a different beast than cached_setup: it does not create a value, just registers a factory. So i don't see how we can unify this. As to a possible resource-factory auto-discovery, i can imagine it to work with introducing a marker:: # example content in a test module or in a conftest.py file @pytest.mark.resourcefactory("db", scope=pytest.Class) def myfactory(name, node): # factory called once per each requesting class (methods # on this class will share the returned value) this declaration would trigger a register_factory("db", myfactory) call. If we want to extend this to parametrization (multiple db factories) we probably need something like this:: @pytest.mark.resourcefactory("db", scope=pytest.Class, multi=True) def make_db_factories(name, node): factoryfuncs = [compute list of factory funcs] return factoryfuncs This would be called at collection time and the scope and the number of to-be-created values would be known in advance. It's basically equivalent to a classnode.register_factory([list of factory funcs]) call. (we could auto-magically recognize yield-generating functions but i'd like to avoid it). To go the full circle, the signature of factory functions could rather accept a "request" object instead of (name, node). Actually today, a request object has this internal state anyway. pytest_funcarg__ would thus only look slighly special in that it skips the marker and has a fixed scope of "pytest.Function". Hope this thought train makes some sense :) holger

2 2

[py-dev] Resource providers
by Floris Bruynooghe June 26, 2012

June 26, 2012

Hi Holger, everyone, Yesterday a resource provider API was considered on IRC, unfortunately I have no logs and forgot the details already. But today I remembered a, possibly invalid, use case which might want to benefit form this: occasionally I wish it was possible to dynamically create funcarg objects. The concrete example I have now is that it could be nice in pytest-django to be able to request e.g. "Users" which is a model class used to access the User table in the database. Currently this is only possible by someone explicitly defining pytest_funcarg__Users, but Django allows you to dynamically look up all the models in the database so there is no reason this can't be build automatically. I think this is what the API you proposed was for, but as I said I can't remember the details. And in this case I might be less enthusiastic in postponing it's implementation to a later release ;-) Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

2 7

[py-dev] Storing terminal width in py.test config object
by Floris Bruynooghe June 26, 2012

June 26, 2012

Hi, An annoyance of the pytest_assertrepr_compare hook is that it can not normally access the terminal width since usually it is called while stdout and stderr are being captured which breaks py.io.get_terminal_width(). Since I think it is fairly rare to get a changing terminal size during a py.test run I would propose py.test stores the terminal width on it's config object which would solve that annoyance. Would this be a reasonable thing to do? Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

3 7

[py-dev] new cache and pep8 pytest plugin releases
by holger krekel June 21, 2012

June 21, 2012

i just released two new plugins: * pytest-cache-0.9 (initial) for easy caching of values across test runs and a new --lf option to rerun the failing tests of a previous run. Install, basic example and API (for use by other plugins) is here: http://packages.python.org/pytest-cache/readme.html * pytest-pep8-1.0 a flexible pep8 checker which allows to keep your project PEP8 compliant with your choice of ignore-options on a per-file basis. It avoids checking files that haven't changed. Examples and docs at: http://pypi.python.org/pypi/pytest-pep8 have fun, holger

1 1

[py-dev] pytest-timeout 0.2
by Floris Bruynooghe June 20, 2012

June 20, 2012

Hi all, I've made a second release of the pytest-timeout plugin for py.test which can time out long running tests. This release includes a number of suggestions made on this list, major changes include: * Fixed the activation problem * Set timeout using configuration file * Add a timeout marker to modify timeout of one item * The marker can also choose the method (signal/thread) * Renamed --nosigalrm to --timeout_method to future proof adding of eventlet and gevent timeout methods * Works on python 3, tested on 2.6, 2.7 and 3.2 Not yet done: * Automatic enabling of the plugin, you still need to enable it on the command line or configuration file before you can use the marker. This was probably a bad idea but I felt bad about stealing a marker by default. * eventlet and gevent timeouts As before the release is on pypi: http://pypi.python.org/pypi/pytest-timeout and the development repository and issue tracker on bitbucket: https://bitbucket.org/flub/pytest-timeout/ I'd be pleased to receive any further feedback you may have. Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

3 8

[py-dev] intermittent bugs in pytest/python on osx lion ?
by Ian Miers June 17, 2012

June 17, 2012

Hi, I just started using pytest. It's lovely. The TLDR on this is we are getting intermittent non reproducible errors that change and sometimes disappear between test runs like the following : > import os.path E TypeError: __import__() argument 1 must be str without null bytes, not str Longer version: We've been getting buggy results out of test runs on OSX Lion with python 3.2.3 and py.test version 2.2.4. Specifically we've been getting what appear to be false-positive test failures that change from run to run and cannot be reproduced by running some code ourselves or in the case of doctest manually running python3.2 -m doctest file. Some of these errors will stay around for multiple test runs even after make clean, etc. Some will change from run to run. All of them eventually disappeared temporarily and we got a clean test pass, though how I don't know. Morever, the problems cropped up again. All of this was with no code changes and code known to work on ubuntu and partially manually tested on OSX. Ordinarly I'd say there was something wrong with our code. However, some of the errors are vanishingly unlikely. Claims that modules don't exist when they do and are importable via python3.2 -c "from foo.bar.baz import narf" and such. The most glaringly, however, is this gem: charm/toolbox/pairinggroup.py:3: in <module> > import os.path E TypeError: __import__() argument 1 must be str without null bytes, not str We also got a lovely bug where it appeared __pycache__ was corrupted during test runs. On an initial run, we could import a function from a python c extension. On subsequent runs, it didn't exist. The function was still in the .so file, as shown by nm, however help(module) returned function_name#$@^%#$% Function description. Deleting __pychache__ folders resolved it for the next test run but then it came back. It too disappeared after a couple of test runs never to be seen since. Has anyone seen anyhting like this? Are their known issues on OSX with python ? With pytest? Does anyone have any idea how I might get a better idea whats going on? As an addendum, the latest error I am getting is now : > test = [hashFn(struct.pack(">%dsI" % (len(seed)), seed, i)) for i in ran] E UnicodeEncodeError: 'ascii' codec can't encode character '\x9e' in position 0: ordinal not in range(128 This actually might be in our code,though again it works on ubuntu and at one point on OSX. Given that its a pattern that points to python or pytest doing something to binaries, I'm including it anyway. The project is charm, you can see the code on this branch here via https://github.com/JHUISI/charm/tree/dev. Python was installed via fenc ( I think, its not my box). The errors happened both with python3.2 -m pytest and with python3.2 setup.py test. Though it appears more so with the later. Thanks, Ian

2 5

[py-dev] pytest-pep8-0.9.1: compatibility to pep8-1.3
by holger krekel June 16, 2012

June 16, 2012

I just did a quick release of pytest-pep8, version 0.9.1 which fixes compatibility issues with the recent pep8 package (1.3). See http://pypi.python.org/pypi/pytest-pep8 for more info. best, holger

1 0

Re: [py-dev] detox-0.9: speeding up tox/test runs with NUM(CORES)
by Tres Seaver June 14, 2012

June 14, 2012

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/14/2012 04:16 AM, Marius Gedminas wrote: > On Wed, Jun 13, 2012 at 02:26:07PM +0000, holger krekel wrote: >> I've finally released detox-0.9 to PYPI which brings parallelizing >> tox/test runs to your console, driving the test runner of your >> choice. detox has the same options and invocation as the good old >> one-step-after-the-other tox, the virtualenv-based generic test >> runner (see http://tox.testrun.org). > > It. Is. AWESOME! Agreed. Very noce work indeed. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver(a)palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk/Z8XMACgkQ+gerLs4ltQ7OjwCfQwVdcSz6G3xZr967nW89A2GE 9uAAn0flCcL9wAPtJ1ffP38WJXd0BhQX =N0G6 -----END PGP SIGNATURE-----

1 0