[py-dev] py.test : setup / teardown at the directory level
Hi, I was wondering if there is a possibility to have or emulate setup/teardown method for an entire directory of tests ? I have an expensive setup/teardown method, which I share through about 10 modules, each with about 20 tests. I would save time if I could do the setup/teardown just for that directory. cheers, Philippe
Hi Philippe, On Tue, Apr 28, 2009 at 13:37 +0200, Philippe Fremy wrote:
Hi,
I was wondering if there is a possibility to have or emulate setup/teardown method for an entire directory of tests ?
I have an expensive setup/teardown method, which I share through about 10 modules, each with about 20 tests. I would save time if I could do the setup/teardown just for that directory.
do you have a setup_module()/teardown_module() pair that you import into all your tests? Do you essentially want to setup the test resource only once per test-run? cheers, holger
cheers,
Philippe
_______________________________________________ py-dev mailing list py-dev@codespeak.net http://codespeak.net/mailman/listinfo/py-dev
-- Metaprogramming, Python, Testing: http://tetamap.wordpress.com Python, PyPy, pytest contracting: http://merlinux.eu
holger krekel wrote:
Hi Philippe,
Hi Holger,
On Tue, Apr 28, 2009 at 13:37 +0200, Philippe Fremy wrote:
Hi,
I was wondering if there is a possibility to have or emulate setup/teardown method for an entire directory of tests ?
I have an expensive setup/teardown method, which I share through about 10 modules, each with about 20 tests. I would save time if I could do the setup/teardown just for that directory.
do you have a setup_module()/teardown_module() pair that you import into all your tests?
Exactly.
Do you essentially want to setup the test resource only once per test-run?
Once per session would be actually sufficient for this specific case today, but the need is once per test directory. To be a bit more explicit, in my current software development, I have both unit tests for functionality in App1 that I develop myself, and functional tests for a wrapper around App2, which is an external application : base + app1 + tests + many unit tests + app2_wrapper + external_app2 + tests + many functional tests for the app2 wrapper, requiring to setup a resource for app2 to run. I launch py.test from the base directory. I have an expensive setup/teardown for the functional tests of the app2 wrapper, which I would like to share at the app2_wrapper/tests directory level. Ideally, if I run only tests from app1, I don't run that expensive setup/teardown. cheers, Philippe
Hi Philippe, your described use case makes lots of sense to me. I coded an example which i hope fits it. It uses the new "local plugins" (i.e. plugins defined in a conftest.py) and funcargs, if you don't know about them yet i hope this is good to skim/read first: http://codespeak.net/py/trunk/test/funcargs.html Here is the example: http://bitbucket.org/hpk42/py-trunk/src/tip/example/funcarg/lazysetup/ using py-trunk (probably also works with the 1.0.0b1, haven't checked) in the lazysetup directory you can now do py.test sub1 # will wait 5 seconds because test # functions access the setup defined in # conftest.py py.test sub2 # will immediately run as the "setup" # funcarg is not requested The idea for this conftest.py implementation is simple: setup the funcarg when first needed and only tear it down when the test process exits. does this make sense to you? feel free to play around and ask questions - I'd then put the above example into the "tutorial" example section of the funcarg doc. One advantage of the above approach is that you do not need to do anything in your test modules anymore (no boilerplate importing of setup_module etc.) than requesting the object you want to setup. cheers, holger On Tue, Apr 28, 2009 at 17:09 +0200, Philippe Fremy wrote:
holger krekel wrote:
Hi Philippe,
Hi Holger,
On Tue, Apr 28, 2009 at 13:37 +0200, Philippe Fremy wrote:
Hi,
I was wondering if there is a possibility to have or emulate setup/teardown method for an entire directory of tests ?
I have an expensive setup/teardown method, which I share through about 10 modules, each with about 20 tests. I would save time if I could do the setup/teardown just for that directory.
do you have a setup_module()/teardown_module() pair that you import into all your tests?
Exactly.
Do you essentially want to setup the test resource only once per test-run?
Once per session would be actually sufficient for this specific case today, but the need is once per test directory.
To be a bit more explicit, in my current software development, I have both unit tests for functionality in App1 that I develop myself, and functional tests for a wrapper around App2, which is an external application :
base + app1 + tests + many unit tests + app2_wrapper + external_app2 + tests + many functional tests for the app2 wrapper, requiring to setup
a resource for app2 to run.
I launch py.test from the base directory.
I have an expensive setup/teardown for the functional tests of the app2 wrapper, which I would like to share at the app2_wrapper/tests directory level.
Ideally, if I run only tests from app1, I don't run that expensive setup/teardown.
cheers,
Philippe
_______________________________________________ py-dev mailing list py-dev@codespeak.net http://codespeak.net/mailman/listinfo/py-dev
-- Metaprogramming, Python, Testing: http://tetamap.wordpress.com Python, PyPy, pytest contracting: http://merlinux.eu
Hi Holger, Thanks for looking into this. I am quite busy until mid next week so it will take me a few days before I get back to you. cheers, Philippe holger krekel wrote:
Hi Philippe,
your described use case makes lots of sense to me.
I coded an example which i hope fits it.
It uses the new "local plugins" (i.e. plugins defined in a conftest.py) and funcargs, if you don't know about them yet i hope this is good to skim/read first: http://codespeak.net/py/trunk/test/funcargs.html
Here is the example:
http://bitbucket.org/hpk42/py-trunk/src/tip/example/funcarg/lazysetup/
using py-trunk (probably also works with the 1.0.0b1, haven't checked) in the lazysetup directory you can now do
py.test sub1 # will wait 5 seconds because test # functions access the setup defined in # conftest.py
py.test sub2 # will immediately run as the "setup" # funcarg is not requested
The idea for this conftest.py implementation is simple: setup the funcarg when first needed and only tear it down when the test process exits.
does this make sense to you? feel free to play around and ask questions - I'd then put the above example into the "tutorial" example section of the funcarg doc.
One advantage of the above approach is that you do not need to do anything in your test modules anymore (no boilerplate importing of setup_module etc.) than requesting the object you want to setup.
cheers, holger
On Tue, Apr 28, 2009 at 17:09 +0200, Philippe Fremy wrote:
holger krekel wrote:
Hi Philippe, Hi Holger,
On Tue, Apr 28, 2009 at 13:37 +0200, Philippe Fremy wrote:
Hi,
I was wondering if there is a possibility to have or emulate setup/teardown method for an entire directory of tests ?
I have an expensive setup/teardown method, which I share through about 10 modules, each with about 20 tests. I would save time if I could do the setup/teardown just for that directory. do you have a setup_module()/teardown_module() pair that you import into all your tests? Exactly.
Do you essentially want to setup the test resource only once per test-run? Once per session would be actually sufficient for this specific case today, but the need is once per test directory.
To be a bit more explicit, in my current software development, I have both unit tests for functionality in App1 that I develop myself, and functional tests for a wrapper around App2, which is an external application :
base + app1 + tests + many unit tests + app2_wrapper + external_app2 + tests + many functional tests for the app2 wrapper, requiring to setup
a resource for app2 to run.
I launch py.test from the base directory.
I have an expensive setup/teardown for the functional tests of the app2 wrapper, which I would like to share at the app2_wrapper/tests directory level.
Ideally, if I run only tests from app1, I don't run that expensive setup/teardown.
cheers,
Philippe
_______________________________________________ py-dev mailing list py-dev@codespeak.net http://codespeak.net/mailman/listinfo/py-dev
holger krekel wrote:
Hi Philippe,
your described use case makes lots of sense to me.
I coded an example which i hope fits it.
It uses the new "local plugins" (i.e. plugins defined in a conftest.py) and funcargs, if you don't know about them yet i hope this is good to skim/read first: http://codespeak.net/py/trunk/test/funcargs.html
Here is the example:
http://bitbucket.org/hpk42/py-trunk/src/tip/example/funcarg/lazysetup/
using py-trunk (probably also works with the 1.0.0b1, haven't checked) in the lazysetup directory you can now do
py.test sub1 # will wait 5 seconds because test # functions access the setup defined in # conftest.py
py.test sub2 # will immediately run as the "setup" # funcarg is not requested
The idea for this conftest.py implementation is simple: setup the funcarg when first needed and only tear it down when the test process exits.
does this make sense to you? feel free to play around and ask questions - I'd then put the above example into the "tutorial" example section of the funcarg doc.
One advantage of the above approach is that you do not need to do anything in your test modules anymore (no boilerplate importing of setup_module etc.) than requesting the object you want to setup.
I am reviving this old thread. Honestly, I haven't tried the funcargs based solution that you propose. The reasons are : - I would really prefer to have setup/teardown at directory level and your solution is more per-session level - I don't like the idea of modifying 100 tests just to get a setup/teardown effect - I still find funcargs a bit cumbersome as explained in my other mail. I had a quick look at the plugin architecture to see if I could implement an equivalent of setup/teardown at directory level, but I don't think it's possible. Can you consider this as a feature request ? cheers, Philippe
Hi Philippe, On Wed, Oct 28, 2009 at 10:09 +0100, Philippe Fremy wrote:
holger krekel wrote:
Hi Philippe,
your described use case makes lots of sense to me.
I coded an example which i hope fits it.
It uses the new "local plugins" (i.e. plugins defined in a conftest.py) and funcargs, if you don't know about them yet i hope this is good to skim/read first: http://codespeak.net/py/trunk/test/funcargs.html
Here is the example:
http://bitbucket.org/hpk42/py-trunk/src/tip/example/funcarg/lazysetup/
using py-trunk (probably also works with the 1.0.0b1, haven't checked) in the lazysetup directory you can now do
py.test sub1 # will wait 5 seconds because test # functions access the setup defined in # conftest.py
py.test sub2 # will immediately run as the "setup" # funcarg is not requested
The idea for this conftest.py implementation is simple: setup the funcarg when first needed and only tear it down when the test process exits.
does this make sense to you? feel free to play around and ask questions - I'd then put the above example into the "tutorial" example section of the funcarg doc.
One advantage of the above approach is that you do not need to do anything in your test modules anymore (no boilerplate importing of setup_module etc.) than requesting the object you want to setup.
I am reviving this old thread.
Honestly, I haven't tried the funcargs based solution that you propose. The reasons are : - I would really prefer to have setup/teardown at directory level and your solution is more per-session level
true, that's currently the case.
- I don't like the idea of modifying 100 tests just to get a setup/teardown effect
Sure. i am thinking about introducing a general "pytest_pyfunc_setup" hook that you could define at project, directory, module or class level and that would be called for setup of each function.
- I still find funcargs a bit cumbersome as explained in my other mail.
I had a quick look at the plugin architecture to see if I could implement an equivalent of setup/teardown at directory level, but I don't think it's possible.
Whatever is called for "directory" setup could live in a conftest.py file. The question is how to transfer any "directory" setup state to a module. At first i thought one could write down: # ./conftest.py def setup_module(mod): # will be called for all modules in/below the dir mod.something = "value" but one would expect this to be called for each module and not just once for a whole directory. Now one could perform some directory-level caching but one conceptual issue remains: values would be "magically" injected into the test modules. Do you have ideas about how you'd like the API to work?
Can you consider this as a feature request ?
i'd like to get an idea on how this could work at all conceptually. best, holger
On Mon, 2010-04-12 at 12:24 +0200, Philippe Fremy wrote:
That thread is 6 months old, but I see that it is still relevant.
holger krekel wrote:
Hi Philippe,
On Wed, Oct 28, 2009 at 10:09 +0100, Philippe Fremy wrote:
I had a quick look at the plugin architecture to see if I could implement an equivalent of setup/teardown at directory level, but I don't think it's possible.
Whatever is called for "directory" setup could live in a conftest.py file. The question is how to transfer any "directory" setup state to a module. At first i thought one could write down:
# ./conftest.py def setup_module(mod): # will be called for all modules in/below the dir mod.something = "value"
but one would expect this to be called for each module and not just once for a whole directory. Now one could perform some directory-level caching but one conceptual issue remains: values would be "magically" injected into the test modules.
Do you have ideas about how you'd like the API to work?
Yes, I now know how I want to transfer the value. I would like to use the py.test namespace.
Today, only pytest_namespace() is available to setup an initial namespace. I would like to see a pytest_update_namespace() to pass a dict with key/values to put in the namespace.
cheers,
Philippe
How about more control over funcargs in particular a) caching on various levels of packages b) exposure of all currently usable funcargs in the py.test namespace -- Ronny
Hi Philippe, Ronny, On Tue, Apr 13, 2010 at 14:19 +0200, Ronny Pfannschmidt wrote:
On Mon, 2010-04-12 at 12:24 +0200, Philippe Fremy wrote:
That thread is 6 months old, but I see that it is still relevant.
holger krekel wrote:
Hi Philippe,
On Wed, Oct 28, 2009 at 10:09 +0100, Philippe Fremy wrote:
I had a quick look at the plugin architecture to see if I could implement an equivalent of setup/teardown at directory level, but I don't think it's possible.
Whatever is called for "directory" setup could live in a conftest.py file. The question is how to transfer any "directory" setup state to a module. At first i thought one could write down:
# ./conftest.py def setup_module(mod): # will be called for all modules in/below the dir mod.something = "value"
but one would expect this to be called for each module and not just once for a whole directory. Now one could perform some directory-level caching but one conceptual issue remains: values would be "magically" injected into the test modules.
Do you have ideas about how you'd like the API to work?
Yes, I now know how I want to transfer the value. I would like to use the py.test namespace.
Today, only pytest_namespace() is available to setup an initial namespace. I would like to see a pytest_update_namespace() to pass a dict with key/values to put in the namespace.
cheers,
Philippe
How about more control over funcargs
in particular
a) caching on various levels of packages b) exposure of all currently usable funcargs in the py.test namespace
Well, funcargs are well defined when used/requested from a test function. Having them globally available blurs the lines a lot and convolutes the py.test namespace. and what would e.g. py.test.tmpdir even be? would it be different when called from a test func, the various setup functions or used at import time etc.? Getting back to Philippe's goal of having per-directory cached values exposed to test functions i wonder if we could a) introduce a directory scope for funcarg-caching b) allow (a subset of) funcargs for setup functions Given this we could write down something like: # ./test_module.py def setup_module(mod, appserver): mod.appserver = appserver # ./conftest.py def pytest_funcarg__appserver(request): return request.cached_setup( setup=lambda: MyAppServer(...), teardown=lambda x: x.shutdown(), scope="directory" ) However, i am not sure if allowing funcargs in setup functions is sane. Also i am not thrilled about encouraging the use of global variables for transfering information. Maybe implementing a "directory" scope for cached_setup would go a long way. Any test function could then directly use the 'appserver' object as a function argument and rely on its per-directory creation/teardown. best, holger
participants (3)
-
holger krekel -
Philippe Fremy -
Ronny Pfannschmidt