Hi Floris, On Sat, Mar 17, 2012 at 18:31 +0100, Floris Bruynooghe wrote:

I've made a second release of the pytest-timeout plugin for py.test which can time out long running tests. This release includes a number of suggestions made on this list, major changes include:

* Fixed the activation problem * Set timeout using configuration file * Add a timeout marker to modify timeout of one item * The marker can also choose the method (signal/thread) * Renamed --nosigalrm to --timeout_method to future proof adding of eventlet and gevent timeout methods * Works on python 3, tested on 2.6, 2.7 and 3.2

Not yet done:

* Automatic enabling of the plugin, you still need to enable it on the command line or configuration file before you can use the marker. This was probably a bad idea but I felt bad about stealing a marker by default.

I went ahead and created a test function with @pytest.mark.timeout(1) def test_hello(): ... but the timeout was not honoured. Then i skimmed the docs :) added "timeout_method = signal" to my ini-file and ran, still not honoured. Then i figured i need to set some dummy "timeout = 10" in the ini - and now i get the proper timeout of 1 second. I understand the hesitance to grab a general name like "timeout" but then again installing "pytest-timeout" is a deliberate act and it grabbing the "timeout" marker is not surprising IMO. So i'd kindly encourage you to go for it. I wonder btw. if the output of "--markers" should be merged with "--help". The latter would get yet longer but then again it's nice to have all the info at a fingertip. Another feedback item: "@pytest.mark.timeout(5, 'signal')" ought to work. It's slightly awkward because of the marker args/kwargs API but it's expected from a pure user perspective i think. Moreover i'd eventually like to include the timeout plugin in pytest core. It's an important feature for functional testing.

* eventlet and gevent timeouts

Here is what i did for eventlet (only accessing the decorator here): https://bitbucket.org/hpk42/detox/src/f9f8c0107cc1/tests/conftest.py#cl-108 cheers & thanks, holger

As before the release is on pypi: http://pypi.python.org/pypi/pytest-timeout and the development repository and issue tracker on bitbucket: https://bitbucket.org/flub/pytest-timeout/

I'd be pleased to receive any further feedback you may have.

Floris

-- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org _______________________________________________ py-dev mailing list py-dev@codespeak.net http://codespeak.net/mailman/listinfo/py-dev

To have this part of the core pytest would be great (even unit tests can hang). It would also be nice to have a consistent behavior between sigalarm on/off. For instance, on Windows, pytest exits on first hang as opposed to *nix where the test is pre-empted and pytest moves on to the next one. Cheers /Laurent -----Original Message----- From: py-dev-bounces@codespeak.net [mailto:py-dev-bounces@codespeak.net] On Behalf Of Floris Bruynooghe Sent: Sunday, March 18, 2012 4:24 PM To: holger krekel; Floris Bruynooghe; py-dev@codespeak.net Subject: Re: [py-dev] pytest-timeout 0.2 Hello Holger, On 18 March 2012 00:16, holger krekel <holger@merlinux.eu> wrote:

I went ahead and created a test function with

   @pytest.mark.timeout(1)    def test_hello():        ...

but the timeout was not honoured.  Then i skimmed the docs :) added "timeout_method = signal" to my ini-file and ran, still not honoured. Then i figured i need to set some dummy "timeout = 10" in the ini - and now i get the proper timeout of 1 second.

Actually timeout=0 would be fine too.

I understand the hesitance to grab a general name like "timeout" but then again installing "pytest-timeout" is a deliberate act and it grabbing the "timeout" marker is not surprising IMO.  So i'd kindly encourage you to go for it. I wonder btw. if the output of "--markers" should be merged with "--help".  The latter would get yet longer but then again it's nice to have all the info at a fingertip.

In fact I had no idea --markers existed, I was considering brining this up but I just hadn't looked around enough! This does make me feel a lot better about using the timeout marker and you've convinced me it's outweighs the stumbling blocks of requiring --timeout=0.

Another feedback item: "@pytest.mark.timeout(5, 'signal')" ought to work. It's slightly awkward because of the marker args/kwargs API but it's expected from a pure user perspective i think.

Yes, I did consider that but then python3 allows keyword only arguments and this seemed a reasonable candidate while the api to make it lends itself to it naturally. But don't see any problem with making it positional as well and am happy to add this.

Moreover i'd eventually like to include the timeout plugin in pytest core.  It's an important feature for functional testing.

That would be nice! But as you implied it probably needs to mature a bit more outside of pytest core first.

...

* eventlet and gevent timeouts

Here is what i did for eventlet (only accessing the decorator here):

https://bitbucket.org/hpk42/detox/src/f9f8c0107cc1/tests/conftest.py#c l-108

Thanks for that hint, hadn't started looking yet but this will save me some experimenting. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org _______________________________________________ py-dev mailing list py-dev@codespeak.net http://codespeak.net/mailman/listinfo/py-dev

I don't want to make this issue overly complicated but here are some scenarios (that we have). Our tests, in some instances, interact with python extensions written in C or C++ (or in some case COM inproc server - yuk) which can be multi-threaded. If the hang occurs in a critical section in the C/C++ domain I am not sure what the timeout effect would be but Assuming that the test is pre-empted in a critical section (of the extension), we would most likely be doomed anyway (mutexes would not be released). Next test would hang, be aborted and so on. In our case, the safest thing to do is to run with xdist -n 1 so then the OS takes care of doing most if not all the cleanup. I am not so intimate with xdist but I was hoping that upon detecting that the "remote process" had died, xdist would restart a new one. In this case pytest ended with: [gw0] win32 -- Python 2.7.2 C:\Python27\python.exe Slave 'gw0' crashed while running 'src/timeout/test_sample.py::TestTimeOut::()::test_hang' Which signaled the end of everything (other tests were not run). This is a something we will have to solve internally (and when we have the solution will be more than happy to contribute it back). We might be able to restart a remote Process on pytest_testnodedown(). We would be however too late to run any restore() and chances are that those might hang as well. More food for thoughts than anything at this point. Floris, thanks for your work. We are not yet in a position where we can help much but we will get to it. /Laurent P.S.: Speaking of xdist, further down the road we are interested in extending xdist (via its hooks) to interface it with OpenStack (dynamic provisioning of VM). Anyone has any use for this or thinking about the same thing? -----Original Message----- From: floris.bruynooghe@gmail.com [mailto:floris.bruynooghe@gmail.com] On Behalf Of Floris Bruynooghe Sent: Monday, March 19, 2012 3:21 PM To: Brack, Laurent P. Cc: holger krekel; py-dev@codespeak.net; Brard, Valentin Subject: Re: [py-dev] pytest-timeout 0.2 On 19 March 2012 16:38, Brack, Laurent P. <lpbrac@dolby.com> wrote:

It would also be nice to have a consistent behavior between sigalarm on/off. For instance, on Windows, pytest exits on first hang as opposed to *nix where the test is pre-empted and pytest moves on to the next one.

I was writing up a response saying I didn't know how to do so, but while doing so I thought it might be possible to emulate sigalrm with a timer thread which fires an other signal. I might investigate the feasibility of such an approach. I do worry about gratuitously using signals however, the simple timer thread with os._exit() may always have to stay as the fail-safe option. If anyone has any other ideas about this please let me know. Regards, Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

Hi Holger, I am not saying there is anything wrong with the way xdist is handling a node going down at this point. the -n 1 was just my attempt to see if the timeout plugin would behave differently when running the test out of process. I think it did, as pytest did not exit the way it does without xdist ... it just ran out of nodes this time and could not carry forward.

Coming to think of it, the xdist plugin might also do "timeout" handling and send a keyboard interrupt from the outside to a running node.

I think this is a good way of handling this. As Floris pointed out, there is always a danger of using signal (right now his marker allows to switch modes on a per test basis which is a good thing).

I am interested in OpenStack but you can you detail a bit more of what you want to achieve?

I will start a separate thread - OpenStack (as it is somewhat unrelated to the timeout subject). /Laurent -----Original Message----- From: holger krekel [mailto:holger@merlinux.eu] Sent: Mon 3/19/2012 11:10 PM To: Brack, Laurent P. Cc: Floris Bruynooghe; holger krekel; py-dev@codespeak.net; Brard, Valentin Subject: Re: [py-dev] pytest-timeout 0.2 Hi Laurent, Floris, On Mon, Mar 19, 2012 at 16:47 -0700, Brack, Laurent P. wrote:

I don't want to make this issue overly complicated but here are some scenarios (that we have).

proper termination and handling of hanging tests is involved. So discussion and hearing use cases makes sense to me.

Our tests, in some instances, interact with python extensions written in C or C++ (or in some case COM inproc server - yuk) which can be multi-threaded.

If the hang occurs in a critical section in the C/C++ domain I am not sure what the timeout effect would be but Assuming that the test is pre-empted in a critical section (of the extension), we would most likely be doomed anyway (mutexes would not be released). Next test would hang, be aborted and so on.

In our case, the safest thing to do is to run with xdist -n 1 so then the OS takes care of doing most if not all the cleanup. I am not so intimate with xdist but I was hoping that upon detecting that the "remote process" had died, xdist would restart a new one.

It's reasonable to expect this from the xdist plugin, even when you run with "-n N" with N>1.

In this case pytest ended with:

[gw0] win32 -- Python 2.7.2 C:\Python27\python.exe Slave 'gw0' crashed while running 'src/timeout/test_sample.py::TestTimeOut::()::test_hang'

Which signaled the end of everything (other tests were not run). This is a something we will have to solve internally (and when we have the solution will be more than happy to contribute it back). We might be able to restart a remote Process on pytest_testnodedown(). We would be however too late to run any restore() and chances are that those might hang as well.

Indeed, no attempt is made to restart a node properly. It remains offline and when run with "-n1" everything dies because no node is left to continue. Not sure how hard it is to try to fix it.

More food for thoughts than anything at this point.

Floris, thanks for your work. We are not yet in a position where we can help much but we will get to it.

Coming to think of it, the xdist plugin might also do "timeout" handling and send a keyboard interrupt from the outside to a running node.

/Laurent

P.S.: Speaking of xdist, further down the road we are interested in extending xdist (via its hooks) to interface it with OpenStack (dynamic provisioning of VM). Anyone has any use for this or thinking about the same thing?

I am interested in OpenStack but you can you detail a bit more of what you want to achieve? best, holger

-----Original Message----- From: floris.bruynooghe@gmail.com [mailto:floris.bruynooghe@gmail.com] On Behalf Of Floris Bruynooghe Sent: Monday, March 19, 2012 3:21 PM To: Brack, Laurent P. Cc: holger krekel; py-dev@codespeak.net; Brard, Valentin Subject: Re: [py-dev] pytest-timeout 0.2

On 19 March 2012 16:38, Brack, Laurent P. <lpbrac@dolby.com> wrote:

...

It would also be nice to have a consistent behavior between sigalarm on/off. For instance, on Windows, pytest exits on first hang as opposed to *nix where the test is pre-empted and pytest moves on to the next one.

I was writing up a response saying I didn't know how to do so, but while doing so I thought it might be possible to emulate sigalrm with a timer thread which fires an other signal. I might investigate the feasibility of such an approach. I do worry about gratuitously using signals however, the simple timer thread with os._exit() may always have to stay as the fail-safe option.

If anyone has any other ideas about this please let me know.

Regards, Floris

-- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org