Re: [py-dev] Decorators and funcargs in py.test
On Mon, 2011-05-30 at 16:23 -0400, Vyacheslav Rafalskiy wrote:
No problem. Here is my (real life) example.
My functional test functions may or may not return for different reasons (like a faulty web application or middleware). I want to declare a fail if it takes more than so many seconds to complete. So I write a decorator run_with_timeout(), which will start the function in a new thread and abandon it after timeout.
This can easily be solved by combining something like pytest.mark('timeout') and a override to pytest_pyfunc_call using it i suppose this could also take a look at extending the "--boxed" mode (which forks for each test and uses subprocesses but doesn’t handle timeouts atm.
A I stated in OP, it is not that I cannot do it. I can and I do. My point is that it makes sense to allow decorators and should not be very difficult (see the example in OP).
personally i am opposed to creating new data conventions for problems that can be solved with plain marks + a hook usually there are 2 reasons people use to decorate tests my opinions for implementing those are a) add arguments + their cleanups -> funcargs please, they are made for that b) use more sophisticated call's -> hooks please, maybe add a bug ticket for empowering one to return a exception-info, so stuff like thread-wrappers can pass that more nicely for failures *i* really think it is wrong to decorate test functions that way and expect stuff to work there are already plenty of mechanisms to change the behavior of pytest test function calling in the desired way, none of those require hacks to pass around argspecs -- Ronny
Thanks, Vyacheslav
On Mon, May 30, 2011 at 3:55 PM, Ronny Pfannschmidt <Ronny.Pfannschmidt@gmx.de> wrote:
Hi,
can you try to explain the usecase those decorators are fulfilling,
there may be a better integrated way using pytest.mark + setup/teardown hooks
-- Ronny
On Mon, 2011-05-30 at 15:17 -0400, Vyacheslav Rafalskiy wrote:
Hi Holger,
I am trying to make decorators work with test functions, which depend on funcargs. As it stands, they don't. Decorated functions lose funcargs. A workaround would be to decorate an internal function like this:
def test_it(funcarg_it): @decorate_it def _test_it(): # test it
_test_it()
This works, but it is not nice. I'd rather wrote a decorator like
def decorate_it(f): def _wrap_it(*args, **kwargs): # wrap f() here
_wrap_it._varnames = _pytest.core.varnames(f) return _wrap_it
and apply it straight to the test function.
After examining the source code I even expected it to just work (magically of course) but it didn't. Do you think it is worthwhile? If so I can enter a feature request.
Thanks, Vyacheslav _______________________________________________ py-dev mailing list py-dev@codespeak.net http://codespeak.net/mailman/listinfo/py-dev
Hi Vyacheslav, hi Ronny, On Mon, May 30, 2011 at 22:38 +0200, Ronny Pfannschmidt wrote:
On Mon, 2011-05-30 at 16:23 -0400, Vyacheslav Rafalskiy wrote:
No problem. Here is my (real life) example.
My functional test functions may or may not return for different reasons (like a faulty web application or middleware). I want to declare a fail if it takes more than so many seconds to complete. So I write a decorator run_with_timeout(), which will start the function in a new thread and abandon it after timeout.
This can easily be solved by combining something like pytest.mark('timeout') and a override to pytest_pyfunc_call using it
I agree. Vyacheslav, if you can't make sense of the "use pytest.mark()" and provide-a-hook suggestion, one of us will be certainly be happy to provide a more complete example and add it to the docs.
i suppose this could also take a look at extending the "--boxed" mode (which forks for each test and uses subprocesses but doesn’t handle timeouts atm.
this is part of the pytest-xdist plugin, however. Question is, if we shouldn't eventually grow timeout-support in core pytest through "alarm" or so. best, holger
A I stated in OP, it is not that I cannot do it. I can and I do. My point is that it makes sense to allow decorators and should not be very difficult (see the example in OP).
personally i am opposed to creating new data conventions for problems that can be solved with plain marks + a hook
usually there are 2 reasons people use to decorate tests
my opinions for implementing those are a) add arguments + their cleanups -> funcargs please, they are made for that b) use more sophisticated call's -> hooks please, maybe add a bug ticket for empowering one to return a exception-info, so stuff like thread-wrappers can pass that more nicely for failures
*i* really think it is wrong to decorate test functions that way and expect stuff to work
there are already plenty of mechanisms to change the behavior of pytest test function calling in the desired way, none of those require hacks to pass around argspecs
-- Ronny
Thanks, Vyacheslav
On Mon, May 30, 2011 at 3:55 PM, Ronny Pfannschmidt <Ronny.Pfannschmidt@gmx.de> wrote:
Hi,
can you try to explain the usecase those decorators are fulfilling,
there may be a better integrated way using pytest.mark + setup/teardown hooks
-- Ronny
On Mon, 2011-05-30 at 15:17 -0400, Vyacheslav Rafalskiy wrote:
Hi Holger,
I am trying to make decorators work with test functions, which depend on funcargs. As it stands, they don't. Decorated functions lose funcargs. A workaround would be to decorate an internal function like this:
def test_it(funcarg_it): @decorate_it def _test_it(): # test it
_test_it()
This works, but it is not nice. I'd rather wrote a decorator like
def decorate_it(f): def _wrap_it(*args, **kwargs): # wrap f() here
_wrap_it._varnames = _pytest.core.varnames(f) return _wrap_it
and apply it straight to the test function.
After examining the source code I even expected it to just work (magically of course) but it didn't. Do you think it is worthwhile? If so I can enter a feature request.
Thanks, Vyacheslav _______________________________________________ py-dev mailing list py-dev@codespeak.net http://codespeak.net/mailman/listinfo/py-dev
_______________________________________________ py-dev mailing list py-dev@codespeak.net http://codespeak.net/mailman/listinfo/py-dev
Gentlemen! Thanks for your input. I really appreciate it. @Ronny 1. I do agree that creating a new data convention is a big deal. What I tried to suggest is using something that already exists in _pytest.core.varnames(). A comment in _pytest.python.getfuncargnames() indicates that the two functions may merge. If they do, and towards the former, my original suggestion will *just work*. 2. Using decorators is a great deal for me. They are powerful and expressive. I understand however the reluctance to open a can of worms here. The replacement using pytest.mark below seems acceptable to me. It still uses the same decorator function, just at a different place. @Holger, Ronny Here is what I came up with: #---------->> in test_1.py @pytest.mark.timeout(10) def test_f1(): # test here #---------->> in conftest.py def pytest_runtest_call(item): if hasattr(item.obj, 'timeout'): timeout = item.obj.timeout.args[0] item.obj = run_with_timeout(timeout)(item.obj) Your comments are welcome. Regards, Vyacheslav On Wed, Jun 1, 2011 at 3:49 AM, holger krekel <holger@merlinux.eu> wrote:
Hi Vyacheslav, hi Ronny,
On Mon, May 30, 2011 at 22:38 +0200, Ronny Pfannschmidt wrote:
On Mon, 2011-05-30 at 16:23 -0400, Vyacheslav Rafalskiy wrote:
No problem. Here is my (real life) example.
My functional test functions may or may not return for different reasons (like a faulty web application or middleware). I want to declare a fail if it takes more than so many seconds to complete. So I write a decorator run_with_timeout(), which will start the function in a new thread and abandon it after timeout.
This can easily be solved by combining something like pytest.mark('timeout') and a override to pytest_pyfunc_call using it
I agree.
Vyacheslav, if you can't make sense of the "use pytest.mark()" and provide-a-hook suggestion, one of us will be certainly be happy to provide a more complete example and add it to the docs.
i suppose this could also take a look at extending the "--boxed" mode (which forks for each test and uses subprocesses but doesn’t handle timeouts atm.
this is part of the pytest-xdist plugin, however. Question is, if we shouldn't eventually grow timeout-support in core pytest through "alarm" or so.
best, holger
A I stated in OP, it is not that I cannot do it. I can and I do. My point is that it makes sense to allow decorators and should not be very difficult (see the example in OP).
personally i am opposed to creating new data conventions for problems that can be solved with plain marks + a hook
usually there are 2 reasons people use to decorate tests
my opinions for implementing those are a) add arguments + their cleanups -> funcargs please, they are made for that b) use more sophisticated call's -> hooks please, maybe add a bug ticket for empowering one to return a exception-info, so stuff like thread-wrappers can pass that more nicely for failures
*i* really think it is wrong to decorate test functions that way and expect stuff to work
there are already plenty of mechanisms to change the behavior of pytest test function calling in the desired way, none of those require hacks to pass around argspecs
-- Ronny
Thanks, Vyacheslav
On Mon, May 30, 2011 at 3:55 PM, Ronny Pfannschmidt <Ronny.Pfannschmidt@gmx.de> wrote:
Hi,
can you try to explain the usecase those decorators are fulfilling,
there may be a better integrated way using pytest.mark + setup/teardown hooks
-- Ronny
On Mon, 2011-05-30 at 15:17 -0400, Vyacheslav Rafalskiy wrote:
Hi Holger,
I am trying to make decorators work with test functions, which depend on funcargs. As it stands, they don't. Decorated functions lose funcargs. A workaround would be to decorate an internal function like this:
def test_it(funcarg_it): @decorate_it def _test_it(): # test it
_test_it()
This works, but it is not nice. I'd rather wrote a decorator like
def decorate_it(f): def _wrap_it(*args, **kwargs): # wrap f() here
_wrap_it._varnames = _pytest.core.varnames(f) return _wrap_it
and apply it straight to the test function.
After examining the source code I even expected it to just work (magically of course) but it didn't. Do you think it is worthwhile? If so I can enter a feature request.
Thanks, Vyacheslav _______________________________________________ py-dev mailing list py-dev@codespeak.net http://codespeak.net/mailman/listinfo/py-dev
_______________________________________________ py-dev mailing list py-dev@codespeak.net http://codespeak.net/mailman/listinfo/py-dev
On Wed, Jun 01, 2011 at 12:31 -0400, Vyacheslav Rafalskiy wrote:
Gentlemen!
Thanks for your input. I really appreciate it.
@Ronny 1. I do agree that creating a new data convention is a big deal. What I tried to suggest is using something that already exists in _pytest.core.varnames(). A comment in _pytest.python.getfuncargnames() indicates that the two functions may merge. If they do, and towards the former, my original suggestion will *just work*.
2. Using decorators is a great deal for me. They are powerful and expressive. I understand however the reluctance to open a can of worms here. The replacement using pytest.mark below seems acceptable to me. It still uses the same decorator function, just at a different place.
@Holger, Ronny Here is what I came up with:
#---------->> in test_1.py @pytest.mark.timeout(10) def test_f1(): # test here
#---------->> in conftest.py def pytest_runtest_call(item): if hasattr(item.obj, 'timeout'): timeout = item.obj.timeout.args[0] item.obj = run_with_timeout(timeout)(item.obj)
Your comments are welcome.
it's basically ok but there are some bits that could be improved. You are actually implementing the general test item call. Also I am not sure why you assign "item.obj = ...". I'd probably write something like this: @pytest.mark.tryfirst def pytest_pyfunc_call(pyfuncitem, __multicall__): if 'timeout' in pyfuncitem.keywords: timeout = pyfuncitem.keywords['timeout'].args[0] run_with_timeout(timeout)(__multicall__.execute) main differences: * only applies to python test function calls * hook invocation will be "tried first" before other pytest_pyfunc_call hook impls and it will call those other hooks through the "__multicall__" bit which actually represents the ongoing hook call * will call other hook implementations How do you actually implement run_with_timeout, btw? best, holger
#---------->> in test_1.py @pytest.mark.timeout(10) def test_f1(): # test here
#---------->> in conftest.py def pytest_runtest_call(item): if hasattr(item.obj, 'timeout'): timeout = item.obj.timeout.args[0] item.obj = run_with_timeout(timeout)(item.obj)
Your comments are welcome.
it's basically ok but there are some bits that could be improved. You are actually implementing the general test item call. Also I am not sure why you assign "item.obj = ...".
I replace (or so I think) the original test function by the decorated one. It gets called elsewhere.
I'd probably write something like this:
@pytest.mark.tryfirst def pytest_pyfunc_call(pyfuncitem, __multicall__): if 'timeout' in pyfuncitem.keywords: timeout = pyfuncitem.keywords['timeout'].args[0] run_with_timeout(timeout)(__multicall__.execute)
this will take a while to digest
main differences:
* only applies to python test function calls * hook invocation will be "tried first" before other pytest_pyfunc_call hook impls and it will call those other hooks through the "__multicall__" bit which actually represents the ongoing hook call * will call other hook implementations
How do you actually implement run_with_timeout, btw?
def run_with_timeout(timeout=None): def _decorator(f): def _wrapper(*args, **kwargs): def _f(*args, **kwargs): try: _result = f(*args, **kwargs) except Exception as e: thread._exception = e else: thread._result = _result thread = threading.Thread(target=_f, args=args, kwargs=kwargs) thread.daemon = True thread._exception = None thread.start() thread.join(timeout=timeout) if thread.isAlive(): raise RuntimeError('function *%s* exceeded configured timeout of %ss' % (f.__name__, timeout)) if thread._exception is None: return thread._result else: raise thread._exception _wrapper.__name__ = f.__name__ return _wrapper return _decorator
best, holger
Vyacheslav
On Wed, Jun 01, 2011 at 17:16 -0400, Vyacheslav Rafalskiy wrote:
#---------->> in test_1.py @pytest.mark.timeout(10) def test_f1(): # test here
#---------->> in conftest.py def pytest_runtest_call(item): if hasattr(item.obj, 'timeout'): timeout = item.obj.timeout.args[0] item.obj = run_with_timeout(timeout)(item.obj)
Your comments are welcome.
it's basically ok but there are some bits that could be improved. You are actually implementing the general test item call. Also I am not sure why you assign "item.obj = ...".
I replace (or so I think) the original test function by the decorated one. It gets called elsewhere.
ah, of course :)
I'd probably write something like this:
@pytest.mark.tryfirst def pytest_pyfunc_call(pyfuncitem, __multicall__): if 'timeout' in pyfuncitem.keywords: timeout = pyfuncitem.keywords['timeout'].args[0] run_with_timeout(timeout)(__multicall__.execute)
this will take a while to digest
it's actually wrong if run_with_timeout is only decorating but not running the function. I think it makes sense to rather directly call a helper which calls the function (note that __multicall__.execute() will execute the remainder of the hook chain one of which will actually execute the function). Such a helper would look like call_with_timeout(timeout, func) i guess. best, holger
main differences:
* only applies to python test function calls * hook invocation will be "tried first" before other pytest_pyfunc_call hook impls and it will call those other hooks through the "__multicall__" bit which actually represents the ongoing hook call * will call other hook implementations
How do you actually implement run_with_timeout, btw?
def run_with_timeout(timeout=None): def _decorator(f): def _wrapper(*args, **kwargs): def _f(*args, **kwargs): try: _result = f(*args, **kwargs) except Exception as e: thread._exception = e else: thread._result = _result
thread = threading.Thread(target=_f, args=args, kwargs=kwargs) thread.daemon = True thread._exception = None
thread.start() thread.join(timeout=timeout) if thread.isAlive(): raise RuntimeError('function *%s* exceeded configured timeout of %ss' % (f.__name__, timeout)) if thread._exception is None: return thread._result else: raise thread._exception _wrapper.__name__ = f.__name__ return _wrapper return _decorator
best, holger
Vyacheslav
On Thu, Jun 2, 2011 at 12:46 AM, holger krekel <holger@merlinux.eu> wrote:
On Wed, Jun 01, 2011 at 17:16 -0400, Vyacheslav Rafalskiy wrote:
#---------->> in test_1.py @pytest.mark.timeout(10) def test_f1(): # test here
#---------->> in conftest.py def pytest_runtest_call(item): if hasattr(item.obj, 'timeout'): timeout = item.obj.timeout.args[0] item.obj = run_with_timeout(timeout)(item.obj)
Your comments are welcome.
it's basically ok but there are some bits that could be improved. You are actually implementing the general test item call. Also I am not sure why you assign "item.obj = ...".
I replace (or so I think) the original test function by the decorated one. It gets called elsewhere.
ah, of course :)
I'd probably write something like this:
@pytest.mark.tryfirst def pytest_pyfunc_call(pyfuncitem, __multicall__): if 'timeout' in pyfuncitem.keywords: timeout = pyfuncitem.keywords['timeout'].args[0] run_with_timeout(timeout)(__multicall__.execute)
this will take a while to digest
it's actually wrong if run_with_timeout is only decorating but not running the function. I think it makes sense to rather directly call a helper which calls the function (note that __multicall__.execute() will execute the remainder of the hook chain one of which will actually execute the function). Such a helper would look like call_with_timeout(timeout, func) i guess.
To call the function I need to know what arguments to give it to. The following seems to work, but this is just a guess: def pytest_runtest_call(item): if hasattr(item.obj, 'timeout'): timeout = item.obj.timeout.args[0] run_with_timeout(timeout)(item.obj)(**item.funcargs) I still don't quite get your example, specifically the __multicall__.execute part. Btw, the pytest_pyfunc_call() parameters seem to be in the wrong order based on _pytest.python.py and the prototype in _pytest.hookspec.py only lists one parameter. Vyacheslav
On Fri, Jun 03, 2011 at 12:06 -0400, Vyacheslav Rafalskiy wrote:
On Thu, Jun 2, 2011 at 12:46 AM, holger krekel <holger@merlinux.eu> wrote:
On Wed, Jun 01, 2011 at 17:16 -0400, Vyacheslav Rafalskiy wrote:
#---------->> in test_1.py @pytest.mark.timeout(10) def test_f1(): # test here
#---------->> in conftest.py def pytest_runtest_call(item): if hasattr(item.obj, 'timeout'): timeout = item.obj.timeout.args[0] item.obj = run_with_timeout(timeout)(item.obj)
Your comments are welcome.
it's basically ok but there are some bits that could be improved. You are actually implementing the general test item call. Also I am not sure why you assign "item.obj = ...".
I replace (or so I think) the original test function by the decorated one. It gets called elsewhere.
ah, of course :)
I'd probably write something like this:
@pytest.mark.tryfirst def pytest_pyfunc_call(pyfuncitem, __multicall__): if 'timeout' in pyfuncitem.keywords: timeout = pyfuncitem.keywords['timeout'].args[0] run_with_timeout(timeout)(__multicall__.execute)
this will take a while to digest
it's actually wrong if run_with_timeout is only decorating but not running the function. I think it makes sense to rather directly call a helper which calls the function (note that __multicall__.execute() will execute the remainder of the hook chain one of which will actually execute the function). Such a helper would look like call_with_timeout(timeout, func) i guess.
To call the function I need to know what arguments to give it to. The following seems to work, but this is just a guess:
def pytest_runtest_call(item): if hasattr(item.obj, 'timeout'): timeout = item.obj.timeout.args[0] run_with_timeout(timeout)(item.obj)(**item.funcargs)
I still don't quite get your example, specifically the __multicall__.execute part.
A few things that might help you to understand: * There can be multiple hook functions implementing the same hook. * MultiCall instances manage the calling into a list of hook implementation functions. * A hook impl function can use zero or any number of available arguments. i.e. if a hookspec is "pyest_myfunc(x,y,z)" then a hook impl function can just receive only one of them "pytest_myfunc(x)" This slicing is done from the MultiCall class. * the actual call to a hook impl function is always done by passing keyword arguments (i.e. **kwargs) and thus ordering of parameters is irrelevant. * a hook impl function can receive a "__multicall__" parameter which is its managing class and which it can use to call __multicall__.execute() to execute the rest of the hook implementations. * with all this in mind the few lines of code in _pytest.core.MultiCall hopefully make more sense :) best, holger
Btw, the pytest_pyfunc_call() parameters seem to be in the wrong order based on _pytest.python.py and the prototype in _pytest.hookspec.py only lists one parameter.
Vyacheslav
Thanks Holger, It is clearer now. Vyacheslav On Fri, Jun 3, 2011 at 1:38 PM, holger krekel <holger@merlinux.eu> wrote:
On Fri, Jun 03, 2011 at 12:06 -0400, Vyacheslav Rafalskiy wrote:
On Thu, Jun 2, 2011 at 12:46 AM, holger krekel <holger@merlinux.eu> wrote:
On Wed, Jun 01, 2011 at 17:16 -0400, Vyacheslav Rafalskiy wrote:
#---------->> in test_1.py @pytest.mark.timeout(10) def test_f1(): # test here
#---------->> in conftest.py def pytest_runtest_call(item): if hasattr(item.obj, 'timeout'): timeout = item.obj.timeout.args[0] item.obj = run_with_timeout(timeout)(item.obj)
Your comments are welcome.
it's basically ok but there are some bits that could be improved. You are actually implementing the general test item call. Also I am not sure why you assign "item.obj = ...".
I replace (or so I think) the original test function by the decorated one. It gets called elsewhere.
ah, of course :)
I'd probably write something like this:
@pytest.mark.tryfirst def pytest_pyfunc_call(pyfuncitem, __multicall__): if 'timeout' in pyfuncitem.keywords: timeout = pyfuncitem.keywords['timeout'].args[0] run_with_timeout(timeout)(__multicall__.execute)
this will take a while to digest
it's actually wrong if run_with_timeout is only decorating but not running the function. I think it makes sense to rather directly call a helper which calls the function (note that __multicall__.execute() will execute the remainder of the hook chain one of which will actually execute the function). Such a helper would look like call_with_timeout(timeout, func) i guess.
To call the function I need to know what arguments to give it to. The following seems to work, but this is just a guess:
def pytest_runtest_call(item): if hasattr(item.obj, 'timeout'): timeout = item.obj.timeout.args[0] run_with_timeout(timeout)(item.obj)(**item.funcargs)
I still don't quite get your example, specifically the __multicall__.execute part.
A few things that might help you to understand:
* There can be multiple hook functions implementing the same hook. * MultiCall instances manage the calling into a list of hook implementation functions. * A hook impl function can use zero or any number of available arguments. i.e. if a hookspec is "pyest_myfunc(x,y,z)" then a hook impl function can just receive only one of them "pytest_myfunc(x)" This slicing is done from the MultiCall class. * the actual call to a hook impl function is always done by passing keyword arguments (i.e. **kwargs) and thus ordering of parameters is irrelevant. * a hook impl function can receive a "__multicall__" parameter which is its managing class and which it can use to call __multicall__.execute() to execute the rest of the hook implementations. * with all this in mind the few lines of code in _pytest.core.MultiCall hopefully make more sense :)
best, holger
Btw, the pytest_pyfunc_call() parameters seem to be in the wrong order based on _pytest.python.py and the prototype in _pytest.hookspec.py only lists one parameter.
Vyacheslav
participants (3)
-
holger krekel -
Ronny Pfannschmidt -
Vyacheslav Rafalskiy