Hello Laurent, On 28 February 2012 18:08, Brack, Laurent P. <lpbrac@dolby.com> wrote:

Just wrote an infinite loop which never got preempted. I didn't have time to dig any deeper. Does the test need to be executed in a separate process via xdist?

xdist is not needed, in fact I haven't tried it but assume it will behave fine. By default SIGALM will be used which does have it's limitations, the main issue is that it needs a chance to deliver the signal to python code. So depending on how your infinite loop is written this might be an issue and you may have to consider --nosigalrm which will use a timer thread (this is more expensive obviously). Could you show me the test function you where trying out?

Also, currently the time out applies to all tests (and defaults to 5 minutes). I think it would be useful to be able to override the timeout at the test level using markers. If something goes wrong and we have thousands of tests (we use pytest to test embedded systems), this is an awful lot of time to realize that things are DOA.

Yes, I agree with this and it is noted in the TODO file that individual tests should have some control on the timeout and the mechanism used (timer vs sigalrm). But we only have about 10000 tests running nightly with normally none blocking, this plugin was written on the spur of the moment when we had about a dozen hanging and this minimal functionality seemed to do the job. It does assume hanging tests are an exception rather then the norm, certainly evident from the behaviour of --nosigalrm where os._exit() is called on a hanging test. I'm happy to work with you on adding a marker which will fulfil your needs however. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

All right, it picked my curiosity so I looked into it. There was fundamentally nothing wrong with your plugin (it does work) however none of the hooks were being called. I could see the FaultHandlerPlugin class was being instanciated during the pytest_configure hook but then None of the hooks were called afterwards. Somehow (and I don't know why), registering the plugin as 'timeout' instead of 'pytest_timeout' was the cause of the problem, so for it to work on my side, I had to do: config.pluginmanager.register(FaultHandlerPlugin(config), 'pytest_timeout') as opposed to config.pluginmanager.register(FaultHandlerPlugin(config), 'timeout') I also played with the ability to use markers to override the default timeout at the test level (here is an excerpt of your code modified) class FaultHandlerPlugin(object): """The timeout plugin""" def __init__(self, config): self.config = config self._current_timer = None self.cancel = None self._default_timeout = self.config.getvalue('timeout') # Default time outset during cfg - Laurent self._timeout = self._default_timeout # Current timeout value @property def timeout(self): return self._timeout @timeout.setter def timeout(self, value): """ Overrides the current timeout - Laurent """ self._timeout = value def reset_timeout(self): """ Resets the timeout to default config value""" self.timeout = self._default_timeout def pytest_runtest_setup(self, item): # Attempt to retrieve a timeout override from the timeout marker # Example: # @pytest.mark.timeout(3) # def test_something(): # ... try: if isinstance(item, item.Function) and hasattr(item.obj, 'timeout'): self.timeout = getattr(item.obj, 'timeout').args[0] except Exception: pass ... def pytest_runtest_teardown(self): """Cancel the timeout trigger""" # When skipping is raised from a pytest_runtest_setup function # (as is the case when using the pytest.mark.skipif marker) we # may be called without our setup counterpart having been # called. Hence the test for self.cancel. if self.cancel: self.cancel() self.cancel = None self.reset_timeout() # reset the timeout to default config value. Here is a sample of tests "working" with the timeout. import sys import pytest import time def test_time_out(): """ Should timeout only with the default timeout. """ counter = 0 while(1): print "hanging ...", counter counter += 1 time.sleep(1) class TestTimeout(): @pytest.mark.timeout(2) def test_time_out_2(self): """ Timeout should be overriden """ counter = 0 while(1): print "hanging ...", counter counter += 1 time.sleep(1) assert counter <= 3 @pytest.mark.timeout(5) def test_time_out_5(self): """ Timeout should be overriden """ counter = 0 while(1): print "hanging ...", counter counter += 1 time.sleep(1) assert counter <= 6 And the output is (I know know, I should have use funcargs but I was lazy :)) def test_time_out(): """ Should timeout only with the default timeout. """ counter = 0 while(1): print "hanging ...", counter counter += 1

time.sleep(1)

E Failed: Timeout >8s test/test_sample.py:14: Failed _________________________________________________________________________________________ TestTimeout.test_time_out_2 _________________________________________________________________________________________ self = <test_sample.TestTimeout instance at 0x90404ac> @pytest.mark.timeout(2) def test_time_out_2(self): """ Timeout should be overriden """ counter = 0 while(1): print "hanging ...", counter counter += 1

time.sleep(1)

E Failed: Timeout >2s test/test_sample.py:25: Failed ________________________________________________________________________________________ TestTimeout.test_time_out_5 _________________________________________________________________________________________ self = <test_sample.TestTimeout instance at 0x904086c> @pytest.mark.timeout(5) def test_time_out_5(self): """ Timeout should be overriden """ counter = 0 while(1): print "hanging ...", counter counter += 1

time.sleep(1)

E Failed: Timeout >5s Cheers /Laurent -----Original Message----- From: py-dev-bounces@codespeak.net on behalf of Brack, Laurent P. Sent: Wed 3/7/2012 10:36 AM To: Floris Bruynooghe Cc: Brard, Valentin; py-dev@codespeak.net Subject: Re: [py-dev] New plugin: pytest-timeout Hello Floris, Thanks for your prompt reply (and sorry for my delayed one :). I haven't had time to play more with the plugin but my observations (and Valentin in CC) has been that neither method was working for us. I haven't done my homework in terms of investigating this further (but I will next week) however, to answer your question the infinite loop looked like this: ------------------------------------------ import time def test_time_out(): counter = 0 while(1): print "hanging ...", counter counter += 1 time.sleep(1) ------------------------------------------ Neither with or without sigalarm worked (I am running on Ubuntu as I mentioned and Valentine tried it on Windows). Based on the code above, I think that both methods should work in that case. Don't worry about it as I will try to figure out what is going on (unless you can think of something). It is however a very nice addition to pytest, as in early development stages things do have a tendency to hang. Thanks /Laurent -----Original Message----- From: floris.bruynooghe@gmail.com on behalf of Floris Bruynooghe Sent: Wed 2/29/2012 2:31 AM To: Brack, Laurent P. Cc: py-dev@codespeak.net Subject: Re: [py-dev] New plugin: pytest-timeout Hello Laurent, On 28 February 2012 18:08, Brack, Laurent P. <lpbrac@dolby.com> wrote:

Just wrote an infinite loop which never got preempted. I didn't have time to dig any deeper. Does the test need to be executed in a separate process via xdist?

xdist is not needed, in fact I haven't tried it but assume it will behave fine. By default SIGALM will be used which does have it's limitations, the main issue is that it needs a chance to deliver the signal to python code. So depending on how your infinite loop is written this might be an issue and you may have to consider --nosigalrm which will use a timer thread (this is more expensive obviously). Could you show me the test function you where trying out?

Also, currently the time out applies to all tests (and defaults to 5 minutes). I think it would be useful to be able to override the timeout at the test level using markers. If something goes wrong and we have thousands of tests (we use pytest to test embedded systems), this is an awful lot of time to realize that things are DOA.

Yes, I agree with this and it is noted in the TODO file that individual tests should have some control on the timeout and the mechanism used (timer vs sigalrm). But we only have about 10000 tests running nightly with normally none blocking, this plugin was written on the spur of the moment when we had about a dozen hanging and this minimal functionality seemed to do the job. It does assume hanging tests are an exception rather then the norm, certainly evident from the behaviour of --nosigalrm where os._exit() is called on a hanging test. I'm happy to work with you on adding a marker which will fulfil your needs however. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

Please be my guest and use it as we are using your code. Thanks a lot for that plugin by the way. -----Original Message----- From: floris.bruynooghe@gmail.com on behalf of Floris Bruynooghe Sent: Thu 3/8/2012 2:49 PM To: Brack, Laurent P. Cc: Brard, Valentin; py-dev@codespeak.net Subject: Re: [py-dev] New plugin: pytest-timeout Hello Laurent, On 7 March 2012 20:56, Brack, Laurent P. <lpbrac@dolby.com> wrote:

Somehow (and I don't know why), registering the plugin as 'timeout' instead of 'pytest_timeout' was the cause of the problem, so for it to work on my side, I had to do:

config.pluginmanager.register(FaultHandlerPlugin(config), 'pytest_timeout')

Ah, I probably got something wrong with the setuptool entrypoints or something. I'll look into it, thanks for finding this!

I also played with the ability to use markers to override the default timeout at the test level (here is an excerpt of your code modified) [...]

Thanks for this, I'll incorporate the timeout marker if you don't mind. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

Hi, in addition something for eventlet/gevent based timeout objects as well as a ini option to choose the timeout method between signal/thread, gevent, eventlet could be nice best, ronny On 03/07/2012 10:55 PM, holger krekel wrote:

Hi Floris,

thanks for the plugin!

looking at version 0.1 i wonder why you didn't go with always activating the plugin so not requiring any pytest_plugins setting. The default could be "no timeout" or None which could be modified with:

[pytest] timeout = 2.3 # secs

or with a marker. Not using some implicit magic number is anyway a good idea i think.

best, holger

On Sun, Feb 26, 2012 at 23:19 +0000, Floris Bruynooghe wrote:

...

Hi all,

This is to announce the first release of pytest-timeout, a plugin which will interrupt long running (i.e. blocking) tests. This is particularly useful when running the tests on a CI host. When a test is interrupted the stacks of all threads are dumped to stderr, which helps you to locate the reason for the blocking. If the system supports SIGALRM then the test itself is interrupted using pytest.fail() and other tests continue to run, otherwise the stack(s) are dumped to stderr and the process exists immediately.

The plugin is available from the cheeseshop: http://pypi.python.org/pypi/pytest-timeout and the code lives at bitbucket: https://bitbucket.org/flub/pytest-timeout/. Feel free to provide any feedback or report issues.

Regards, Floris

-- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org _______________________________________________

On 03/09/2012 12:17 AM, Floris Bruynooghe wrote:

On 7 March 2012 23:35, Ronny Pfannschmidt<Ronny.Pfannschmidt@gmx.de> wrote:

...

in addition something for eventlet/gevent based timeout objects as well as a ini option to choose the timeout method between signal/thread, gevent, eventlet could be nice

eventlet/gevent timeouts are also a good idea, I'll look into adding these as well.

One issue with this however is that I hate calling import from inside a function (partially because I've looked too close at eventlet's patching code) but unconditionally importing it upfront, in a try-except block, woud be bad for py.test's startup time. Would importing in a function be reasonable in this case? Or am I missing another solution?

Lets go with the assumption that whoever is requesting a gevent/eventlet timeout knows enough of what hes doing (since it would need a config entry) then its reasonable enough to just grab the timeouts in the function wrt monkeypatching, i think its strictly opt-in, and should also be left to the user best, Ronny

Thanks for all the feedback everyone!

Floris

On Thu, Mar 08, 2012 at 23:08 +0000, Floris Bruynooghe wrote:

On 7 March 2012 21:55, holger krekel <holger@merlinux.eu> wrote:

...

or with a marker.  Not using some implicit magic number is anyway a good idea i think.

Not sure what you mean. Do you mean using 0 as saying "no timeout" is a magic number but e.g. None is fine? Essentially anything else other then a positive number is "disable timeout" to me.

just meant "5 seconds" as a default. 0 or None is fine to mean "no timeout" IMHO. best, holger

Regards, Floris

-- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org