[py-dev] New plugin: pytest-timeout

[Brack, Laurent P.]

All right, it picked my curiosity so I looked into it. 

There was fundamentally nothing wrong with your plugin (it does work) however none of the hooks were being called. 
I could see the FaultHandlerPlugin class was being instanciated during the pytest_configure hook but then None of the hooks
were called afterwards. 

Somehow (and I don't know why), registering the plugin as 'timeout' instead of 'pytest_timeout' was the cause of the problem,
so for it to work on my side, I had to do:

config.pluginmanager.register(FaultHandlerPlugin(config), 'pytest_timeout')

as opposed to 

config.pluginmanager.register(FaultHandlerPlugin(config), 'timeout')

I also played with the ability to use markers to override the default timeout at the test level (here is an excerpt of 
your code modified)

class FaultHandlerPlugin(object):
    """The timeout plugin"""

    def __init__(self, config):
        self.config = config
        self._current_timer = None
        self.cancel = None
        self._default_timeout = self.config.getvalue('timeout') # Default time outset during cfg - Laurent
        self._timeout = self._default_timeout                   # Current timeout value

    @property
    def timeout(self):
        return self._timeout
    
    @timeout.setter
    def timeout(self, value):
        """ Overrides the current timeout - Laurent """
        self._timeout = value
        
    def reset_timeout(self):
        """ Resets the timeout to default config value"""
        self.timeout = self._default_timeout 
    

    def pytest_runtest_setup(self, item):

        # Attempt to retrieve a timeout override from the timeout marker
        # Example:
        # @pytest.mark.timeout(3)
        # def test_something():
        #     ...
        try:        
            if isinstance(item, item.Function) and hasattr(item.obj, 'timeout'):
                self.timeout = getattr(item.obj, 'timeout').args[0]               
        except Exception:
            pass

        ...

    def pytest_runtest_teardown(self):
        """Cancel the timeout trigger"""
        # When skipping is raised from a pytest_runtest_setup function
        # (as is the case when using the pytest.mark.skipif marker) we
        # may be called without our setup counterpart having been
        # called.  Hence the test for self.cancel.
        if self.cancel:
            self.cancel()
            self.cancel = None
        self.reset_timeout() # reset the timeout to default config value.

Here is a sample of tests "working" with the timeout.

import sys
import pytest
import time

def test_time_out():
    """ Should timeout only with the default timeout. """
    counter = 0
    while(1):
        print "hanging ...", counter
        counter += 1
        time.sleep(1)
        
class TestTimeout():

    @pytest.mark.timeout(2)    
    def test_time_out_2(self):
        """ Timeout should be overriden """
        counter = 0
        while(1):
            print "hanging ...", counter
            counter += 1
            time.sleep(1)   
            assert counter <= 3         
            
    @pytest.mark.timeout(5)    
    def test_time_out_5(self):
        """ Timeout should be overriden """
        counter = 0
        while(1):
            print "hanging ...", counter
            counter += 1
            time.sleep(1)   
            assert counter <= 6   

And the output is (I know know, I should have use funcargs but I was lazy :))

    def test_time_out():
        """ Should timeout only with the default timeout. """
        counter = 0
        while(1):
            print "hanging ...", counter
            counter += 1
>           time.sleep(1)
E           Failed: Timeout >8s

test/test_sample.py:14: Failed
_________________________________________________________________________________________ 
TestTimeout.test_time_out_2 
_________________________________________________________________________________________

self = <test_sample.TestTimeout instance at 0x90404ac>

    @pytest.mark.timeout(2)
    def test_time_out_2(self):
        """ Timeout should be overriden """
        counter = 0
        while(1):
            print "hanging ...", counter
            counter += 1
>           time.sleep(1)
E           Failed: Timeout >2s

test/test_sample.py:25: Failed
________________________________________________________________________________________ 
TestTimeout.test_time_out_5 
_________________________________________________________________________________________

self = <test_sample.TestTimeout instance at 0x904086c>

    @pytest.mark.timeout(5)
    def test_time_out_5(self):
        """ Timeout should be overriden """
        counter = 0
        while(1):
            print "hanging ...", counter
            counter += 1
>           time.sleep(1)
E           Failed: Timeout >5s



Cheers

/Laurent

-----Original Message-----
From: py-dev-bounces at codespeak.net on behalf of Brack, Laurent P.
Sent: Wed 3/7/2012 10:36 AM
To: Floris Bruynooghe
Cc: Brard, Valentin; py-dev at codespeak.net
Subject: Re: [py-dev] New plugin: pytest-timeout
 
Hello Floris,

Thanks for your prompt reply (and sorry for my delayed one :). 

I haven't had time to play more with the plugin but my observations (and Valentin in CC) has been that neither method was working for us. 

I haven't done my homework in terms of investigating this further (but I will next week) however, to answer your question the infinite loop looked like this:
------------------------------------------
import time

def test_time_out():
    counter = 0
    while(1):
        print "hanging ...", counter
        counter += 1
        time.sleep(1)
------------------------------------------

Neither with or without sigalarm worked (I am running on Ubuntu as I mentioned and Valentine tried it on Windows). 
Based on the code above, I think that both methods should work in that case. Don't worry about it as I will try to figure out 
what is going on (unless you can think of something). It is however a very nice addition to pytest, as in early development 
stages things do have a tendency to hang.

Thanks

/Laurent

-----Original Message-----
From: floris.bruynooghe at gmail.com on behalf of Floris Bruynooghe
Sent: Wed 2/29/2012 2:31 AM
To: Brack, Laurent P.
Cc: py-dev at codespeak.net
Subject: Re: [py-dev] New plugin: pytest-timeout
 
Hello Laurent,

On 28 February 2012 18:08, Brack, Laurent P. <lpbrac at dolby.com> wrote:
> Just wrote an infinite loop which never got preempted. I didn't have time to
> dig any deeper. Does the test need to be executed in a separate process via
> xdist?

xdist is not needed, in fact I haven't tried it but assume it will
behave fine.  By default SIGALM will be used which does have it's
limitations, the main issue is that it needs a chance to deliver the
signal to python code.  So depending on how your infinite loop is
written this might be an issue and you may have to consider
--nosigalrm which will use a timer thread (this is more expensive
obviously).  Could you show me the test function you where trying out?

> Also, currently the time out applies to all tests (and defaults to 5
> minutes). I think it would be useful to be able to override the timeout
> at the test level using markers. If something goes wrong and we have
> thousands of tests (we use pytest to test embedded systems), this is an
> awful lot of time to realize that things are DOA.

Yes, I agree with this and it is noted in the TODO file that
individual tests should have some control on the timeout and the
mechanism used (timer vs sigalrm).  But we only have about 10000 tests
running nightly with normally none blocking, this plugin was written
on the spur of the moment when we had about a dozen hanging and this
minimal functionality seemed to do the job.  It does assume hanging
tests are an exception rather then the norm, certainly evident from
the behaviour of --nosigalrm where os._exit() is called on a hanging
test.

I'm happy to work with you on adding a marker which will fulfil your
needs however.

Regards,
Floris


-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20120307/7f74e040/attachment.html>