From holger at merlinux.eu  Fri Mar  2 15:12:58 2012
From: holger at merlinux.eu (holger krekel)
Date: Fri, 2 Mar 2012 14:12:58 +0000
Subject: [py-dev] XML Parsing Error when esc characters are in traceback
In-Reply-To: <CALL=tm3Op_OBSugFtbks9sm0nHTCCxosW7rgAvLFV9YZ8grWFA@mail.gmail.com>
References: <CALL=tm3Op_OBSugFtbks9sm0nHTCCxosW7rgAvLFV9YZ8grWFA@mail.gmail.com>
Message-ID: <20120302141258.GS26028@merlinux.eu>

Hi Anton,

guess you know it - Ronny opened a bug on this issue and you
might want to follow it. See

https://bitbucket.org/hpk42/pytest/issue/126/junitxml-doesnt-escape-everything

holger

On Wed, Feb 22, 2012 at 13:10 +0200, Anton P wrote:
> Hello all,
> 
> I have the following issue with generated junitxml reports:
> 
> I'm writing test cases which use scapy functions. So I make asserts on
> some packet objects. As I understand scapy uses esc characters for its
> own objects representation, and I can see colorful output on the
> console. But when test is failed, and its traceback went to xml, I
> can't open generated xml due to the error "XML Parsing Error: not
> well-formed" In my case the problem with esc charecter 033 (\x1b).
> I resolved the issue by adding the following code to append_failure
> method in LogXML class:
> longrepr = str(report.longrepr).replace('\x1b', '\\033')
> fail.append(longrepr)
> 
> I think that it will be good if you add some cleanup functions to
> avoid such situations.
> 
> Thank you,
> 
> -Anton
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
> 


From anton7811 at gmail.com  Mon Mar  5 10:56:40 2012
From: anton7811 at gmail.com (Anton P)
Date: Mon, 5 Mar 2012 11:56:40 +0200
Subject: [py-dev] XML Parsing Error when esc characters are in traceback
In-Reply-To: <20120302141258.GS26028@merlinux.eu>
References: <CALL=tm3Op_OBSugFtbks9sm0nHTCCxosW7rgAvLFV9YZ8grWFA@mail.gmail.com>
	<20120302141258.GS26028@merlinux.eu>
Message-ID: <CALL=tm1t=3w=beZQUoOS80UGRXSKU442Xu9zUg+fpVLHh-+UcA@mail.gmail.com>

Thank you!
I've checked that.


2012/3/2 holger krekel <holger at merlinux.eu>:
> Hi Anton,
>
> guess you know it - Ronny opened a bug on this issue and you
> might want to follow it. See
>
> https://bitbucket.org/hpk42/pytest/issue/126/junitxml-doesnt-escape-everything
>
> holger
>
> On Wed, Feb 22, 2012 at 13:10 +0200, Anton P wrote:
>> Hello all,
>>
>> I have the following issue with generated junitxml reports:
>>
>> I'm writing test cases which use scapy functions. So I make asserts on
>> some packet objects. As I understand scapy uses esc characters for its
>> own objects representation, and I can see colorful output on the
>> console. But when test is failed, and its traceback went to xml, I
>> can't open generated xml due to the error "XML Parsing Error: not
>> well-formed" In my case the problem with esc charecter 033 (\x1b).
>> I resolved the issue by adding the following code to append_failure
>> method in LogXML class:
>> longrepr = str(report.longrepr).replace('\x1b', '\\033')
>> fail.append(longrepr)
>>
>> I think that it will be good if you add some cleanup functions to
>> avoid such situations.
>>
>> Thank you,
>>
>> -Anton
>> _______________________________________________
>> py-dev mailing list
>> py-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/py-dev
>>


From lpbrac at dolby.com  Wed Mar  7 19:36:14 2012
From: lpbrac at dolby.com (Brack, Laurent P.)
Date: Wed, 7 Mar 2012 10:36:14 -0800
Subject: [py-dev] New plugin: pytest-timeout
References: <CAAWk_DxA3mh7GmNSRW5_EtGACH4qgeGQW2X4aZpGuiReP2Z6Ng@mail.gmail.com><05BB196AB3DA6C4BBE11AB6C957581FE42A39DB3@sfo-exch-01.dolby.net>
	<CAAWk_DyYqj=v7gvPdFx7mddx4BZbfqEYCDtD_Ofy1rhEuZ=euA@mail.gmail.com>
Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE42A39E96@sfo-exch-01.dolby.net>

Hello Floris,

Thanks for your prompt reply (and sorry for my delayed one :). 

I haven't had time to play more with the plugin but my observations (and Valentin in CC) has been that neither method was working for us. 

I haven't done my homework in terms of investigating this further (but I will next week) however, to answer your question the infinite loop looked like this:
------------------------------------------
import time

def test_time_out():
    counter = 0
    while(1):
        print "hanging ...", counter
        counter += 1
        time.sleep(1)
------------------------------------------

Neither with or without sigalarm worked (I am running on Ubuntu as I mentioned and Valentine tried it on Windows). 
Based on the code above, I think that both methods should work in that case. Don't worry about it as I will try to figure out 
what is going on (unless you can think of something). It is however a very nice addition to pytest, as in early development 
stages things do have a tendency to hang.

Thanks

/Laurent

-----Original Message-----
From: floris.bruynooghe at gmail.com on behalf of Floris Bruynooghe
Sent: Wed 2/29/2012 2:31 AM
To: Brack, Laurent P.
Cc: py-dev at codespeak.net
Subject: Re: [py-dev] New plugin: pytest-timeout
 
Hello Laurent,

On 28 February 2012 18:08, Brack, Laurent P. <lpbrac at dolby.com> wrote:
> Just wrote an infinite loop which never got preempted. I didn't have time to
> dig any deeper. Does the test need to be executed in a separate process via
> xdist?

xdist is not needed, in fact I haven't tried it but assume it will
behave fine.  By default SIGALM will be used which does have it's
limitations, the main issue is that it needs a chance to deliver the
signal to python code.  So depending on how your infinite loop is
written this might be an issue and you may have to consider
--nosigalrm which will use a timer thread (this is more expensive
obviously).  Could you show me the test function you where trying out?

> Also, currently the time out applies to all tests (and defaults to 5
> minutes). I think it would be useful to be able to override the timeout
> at the test level using markers. If something goes wrong and we have
> thousands of tests (we use pytest to test embedded systems), this is an
> awful lot of time to realize that things are DOA.

Yes, I agree with this and it is noted in the TODO file that
individual tests should have some control on the timeout and the
mechanism used (timer vs sigalrm).  But we only have about 10000 tests
running nightly with normally none blocking, this plugin was written
on the spur of the moment when we had about a dozen hanging and this
minimal functionality seemed to do the job.  It does assume hanging
tests are an exception rather then the norm, certainly evident from
the behaviour of --nosigalrm where os._exit() is called on a hanging
test.

I'm happy to work with you on adding a marker which will fulfil your
needs however.

Regards,
Floris


-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20120307/6f7d33a3/attachment.html>

From lpbrac at dolby.com  Wed Mar  7 21:56:13 2012
From: lpbrac at dolby.com (Brack, Laurent P.)
Date: Wed, 7 Mar 2012 12:56:13 -0800
Subject: [py-dev] New plugin: pytest-timeout
References: <CAAWk_DxA3mh7GmNSRW5_EtGACH4qgeGQW2X4aZpGuiReP2Z6Ng@mail.gmail.com><05BB196AB3DA6C4BBE11AB6C957581FE42A39DB3@sfo-exch-01.dolby.net><CAAWk_DyYqj=v7gvPdFx7mddx4BZbfqEYCDtD_Ofy1rhEuZ=euA@mail.gmail.com>
	<05BB196AB3DA6C4BBE11AB6C957581FE42A39E96@sfo-exch-01.dolby.net>
Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE42A39E98@sfo-exch-01.dolby.net>

All right, it picked my curiosity so I looked into it. 

There was fundamentally nothing wrong with your plugin (it does work) however none of the hooks were being called. 
I could see the FaultHandlerPlugin class was being instanciated during the pytest_configure hook but then None of the hooks
were called afterwards. 

Somehow (and I don't know why), registering the plugin as 'timeout' instead of 'pytest_timeout' was the cause of the problem,
so for it to work on my side, I had to do:

config.pluginmanager.register(FaultHandlerPlugin(config), 'pytest_timeout')

as opposed to 

config.pluginmanager.register(FaultHandlerPlugin(config), 'timeout')

I also played with the ability to use markers to override the default timeout at the test level (here is an excerpt of 
your code modified)

class FaultHandlerPlugin(object):
    """The timeout plugin"""

    def __init__(self, config):
        self.config = config
        self._current_timer = None
        self.cancel = None
        self._default_timeout = self.config.getvalue('timeout') # Default time outset during cfg - Laurent
        self._timeout = self._default_timeout                   # Current timeout value

    @property
    def timeout(self):
        return self._timeout
    
    @timeout.setter
    def timeout(self, value):
        """ Overrides the current timeout - Laurent """
        self._timeout = value
        
    def reset_timeout(self):
        """ Resets the timeout to default config value"""
        self.timeout = self._default_timeout 
    

    def pytest_runtest_setup(self, item):

        # Attempt to retrieve a timeout override from the timeout marker
        # Example:
        # @pytest.mark.timeout(3)
        # def test_something():
        #     ...
        try:        
            if isinstance(item, item.Function) and hasattr(item.obj, 'timeout'):
                self.timeout = getattr(item.obj, 'timeout').args[0]               
        except Exception:
            pass

        ...

    def pytest_runtest_teardown(self):
        """Cancel the timeout trigger"""
        # When skipping is raised from a pytest_runtest_setup function
        # (as is the case when using the pytest.mark.skipif marker) we
        # may be called without our setup counterpart having been
        # called.  Hence the test for self.cancel.
        if self.cancel:
            self.cancel()
            self.cancel = None
        self.reset_timeout() # reset the timeout to default config value.

Here is a sample of tests "working" with the timeout.

import sys
import pytest
import time

def test_time_out():
    """ Should timeout only with the default timeout. """
    counter = 0
    while(1):
        print "hanging ...", counter
        counter += 1
        time.sleep(1)
        
class TestTimeout():

    @pytest.mark.timeout(2)    
    def test_time_out_2(self):
        """ Timeout should be overriden """
        counter = 0
        while(1):
            print "hanging ...", counter
            counter += 1
            time.sleep(1)   
            assert counter <= 3         
            
    @pytest.mark.timeout(5)    
    def test_time_out_5(self):
        """ Timeout should be overriden """
        counter = 0
        while(1):
            print "hanging ...", counter
            counter += 1
            time.sleep(1)   
            assert counter <= 6   

And the output is (I know know, I should have use funcargs but I was lazy :))

    def test_time_out():
        """ Should timeout only with the default timeout. """
        counter = 0
        while(1):
            print "hanging ...", counter
            counter += 1
>           time.sleep(1)
E           Failed: Timeout >8s

test/test_sample.py:14: Failed
_________________________________________________________________________________________ 
TestTimeout.test_time_out_2 
_________________________________________________________________________________________

self = <test_sample.TestTimeout instance at 0x90404ac>

    @pytest.mark.timeout(2)
    def test_time_out_2(self):
        """ Timeout should be overriden """
        counter = 0
        while(1):
            print "hanging ...", counter
            counter += 1
>           time.sleep(1)
E           Failed: Timeout >2s

test/test_sample.py:25: Failed
________________________________________________________________________________________ 
TestTimeout.test_time_out_5 
_________________________________________________________________________________________

self = <test_sample.TestTimeout instance at 0x904086c>

    @pytest.mark.timeout(5)
    def test_time_out_5(self):
        """ Timeout should be overriden """
        counter = 0
        while(1):
            print "hanging ...", counter
            counter += 1
>           time.sleep(1)
E           Failed: Timeout >5s



Cheers

/Laurent

-----Original Message-----
From: py-dev-bounces at codespeak.net on behalf of Brack, Laurent P.
Sent: Wed 3/7/2012 10:36 AM
To: Floris Bruynooghe
Cc: Brard, Valentin; py-dev at codespeak.net
Subject: Re: [py-dev] New plugin: pytest-timeout
 
Hello Floris,

Thanks for your prompt reply (and sorry for my delayed one :). 

I haven't had time to play more with the plugin but my observations (and Valentin in CC) has been that neither method was working for us. 

I haven't done my homework in terms of investigating this further (but I will next week) however, to answer your question the infinite loop looked like this:
------------------------------------------
import time

def test_time_out():
    counter = 0
    while(1):
        print "hanging ...", counter
        counter += 1
        time.sleep(1)
------------------------------------------

Neither with or without sigalarm worked (I am running on Ubuntu as I mentioned and Valentine tried it on Windows). 
Based on the code above, I think that both methods should work in that case. Don't worry about it as I will try to figure out 
what is going on (unless you can think of something). It is however a very nice addition to pytest, as in early development 
stages things do have a tendency to hang.

Thanks

/Laurent

-----Original Message-----
From: floris.bruynooghe at gmail.com on behalf of Floris Bruynooghe
Sent: Wed 2/29/2012 2:31 AM
To: Brack, Laurent P.
Cc: py-dev at codespeak.net
Subject: Re: [py-dev] New plugin: pytest-timeout
 
Hello Laurent,

On 28 February 2012 18:08, Brack, Laurent P. <lpbrac at dolby.com> wrote:
> Just wrote an infinite loop which never got preempted. I didn't have time to
> dig any deeper. Does the test need to be executed in a separate process via
> xdist?

xdist is not needed, in fact I haven't tried it but assume it will
behave fine.  By default SIGALM will be used which does have it's
limitations, the main issue is that it needs a chance to deliver the
signal to python code.  So depending on how your infinite loop is
written this might be an issue and you may have to consider
--nosigalrm which will use a timer thread (this is more expensive
obviously).  Could you show me the test function you where trying out?

> Also, currently the time out applies to all tests (and defaults to 5
> minutes). I think it would be useful to be able to override the timeout
> at the test level using markers. If something goes wrong and we have
> thousands of tests (we use pytest to test embedded systems), this is an
> awful lot of time to realize that things are DOA.

Yes, I agree with this and it is noted in the TODO file that
individual tests should have some control on the timeout and the
mechanism used (timer vs sigalrm).  But we only have about 10000 tests
running nightly with normally none blocking, this plugin was written
on the spur of the moment when we had about a dozen hanging and this
minimal functionality seemed to do the job.  It does assume hanging
tests are an exception rather then the norm, certainly evident from
the behaviour of --nosigalrm where os._exit() is called on a hanging
test.

I'm happy to work with you on adding a marker which will fulfil your
needs however.

Regards,
Floris


-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20120307/7f74e040/attachment.html>

From holger at merlinux.eu  Wed Mar  7 22:55:49 2012
From: holger at merlinux.eu (holger krekel)
Date: Wed, 7 Mar 2012 21:55:49 +0000
Subject: [py-dev] New plugin: pytest-timeout
In-Reply-To: <CAAWk_DxA3mh7GmNSRW5_EtGACH4qgeGQW2X4aZpGuiReP2Z6Ng@mail.gmail.com>
References: <CAAWk_DxA3mh7GmNSRW5_EtGACH4qgeGQW2X4aZpGuiReP2Z6Ng@mail.gmail.com>
Message-ID: <20120307215549.GH26028@merlinux.eu>

Hi Floris,

thanks for the plugin!

looking at version 0.1 i wonder why you didn't go with always activating
the plugin so not requiring any pytest_plugins setting.  The default
could be "no timeout" or None which could be modified with:

    [pytest]
    timeout = 2.3 # secs

or with a marker.  Not using some implicit magic number is anyway a good
idea i think.

best,
holger

On Sun, Feb 26, 2012 at 23:19 +0000, Floris Bruynooghe wrote:
> Hi all,
> 
> This is to announce the first release of pytest-timeout, a plugin
> which will interrupt long running (i.e. blocking) tests.  This is
> particularly useful when running the tests on a CI host. When a test
> is interrupted the stacks of all threads are dumped to stderr, which
> helps you to locate the reason for the blocking.  If the system
> supports SIGALRM then the test itself is interrupted using
> pytest.fail() and other tests continue to run, otherwise the stack(s)
> are dumped to stderr and the process exists immediately.
> 
> The plugin is available from the cheeseshop:
> http://pypi.python.org/pypi/pytest-timeout and the code lives at
> bitbucket: https://bitbucket.org/flub/pytest-timeout/.  Feel free to
> provide any feedback or report issues.
> 
> Regards,
> Floris
> 
> -- 
> Debian GNU/Linux -- The Power of Freedom
> www.debian.org | www.gnu.org | www.kernel.org
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
> 


From Ronny.Pfannschmidt at gmx.de  Thu Mar  8 00:35:08 2012
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Thu, 08 Mar 2012 00:35:08 +0100
Subject: [py-dev] New plugin: pytest-timeout
In-Reply-To: <20120307215549.GH26028@merlinux.eu>
References: <CAAWk_DxA3mh7GmNSRW5_EtGACH4qgeGQW2X4aZpGuiReP2Z6Ng@mail.gmail.com>
	<20120307215549.GH26028@merlinux.eu>
Message-ID: <4F57F0AC.7000603@gmx.de>

Hi,

in addition something for eventlet/gevent based timeout objects
as well as a ini option to choose the timeout method between 
signal/thread, gevent, eventlet could be nice

best,
ronny

On 03/07/2012 10:55 PM, holger krekel wrote:
> Hi Floris,
>
> thanks for the plugin!
>
> looking at version 0.1 i wonder why you didn't go with always activating
> the plugin so not requiring any pytest_plugins setting.  The default
> could be "no timeout" or None which could be modified with:
>
>      [pytest]
>      timeout = 2.3 # secs
>
> or with a marker.  Not using some implicit magic number is anyway a good
> idea i think.
>
> best,
> holger
>
> On Sun, Feb 26, 2012 at 23:19 +0000, Floris Bruynooghe wrote:
>> Hi all,
>>
>> This is to announce the first release of pytest-timeout, a plugin
>> which will interrupt long running (i.e. blocking) tests.  This is
>> particularly useful when running the tests on a CI host. When a test
>> is interrupted the stacks of all threads are dumped to stderr, which
>> helps you to locate the reason for the blocking.  If the system
>> supports SIGALRM then the test itself is interrupted using
>> pytest.fail() and other tests continue to run, otherwise the stack(s)
>> are dumped to stderr and the process exists immediately.
>>
>> The plugin is available from the cheeseshop:
>> http://pypi.python.org/pypi/pytest-timeout and the code lives at
>> bitbucket: https://bitbucket.org/flub/pytest-timeout/.  Feel free to
>> provide any feedback or report issues.
>>
>> Regards,
>> Floris
>>
>> --
>> Debian GNU/Linux -- The Power of Freedom
>> www.debian.org | www.gnu.org | www.kernel.org
>> _______________________________________________


From flub at devork.be  Thu Mar  8 23:49:46 2012
From: flub at devork.be (Floris Bruynooghe)
Date: Thu, 8 Mar 2012 22:49:46 +0000
Subject: [py-dev] New plugin: pytest-timeout
In-Reply-To: <05BB196AB3DA6C4BBE11AB6C957581FE42A39E98@sfo-exch-01.dolby.net>
References: <CAAWk_DxA3mh7GmNSRW5_EtGACH4qgeGQW2X4aZpGuiReP2Z6Ng@mail.gmail.com>
	<05BB196AB3DA6C4BBE11AB6C957581FE42A39DB3@sfo-exch-01.dolby.net>
	<CAAWk_DyYqj=v7gvPdFx7mddx4BZbfqEYCDtD_Ofy1rhEuZ=euA@mail.gmail.com>
	<05BB196AB3DA6C4BBE11AB6C957581FE42A39E96@sfo-exch-01.dolby.net>
	<05BB196AB3DA6C4BBE11AB6C957581FE42A39E98@sfo-exch-01.dolby.net>
Message-ID: <CAAWk_DxhuqMYV-Rp-B6mxSH3gKVFUcwdhNGEXgQKko4PXgjNJA@mail.gmail.com>

Hello Laurent,

On 7 March 2012 20:56, Brack, Laurent P. <lpbrac at dolby.com> wrote:
> Somehow (and I don't know why), registering the plugin as 'timeout' instead
> of 'pytest_timeout' was the cause of the problem,
> so for it to work on my side, I had to do:
>
> config.pluginmanager.register(FaultHandlerPlugin(config), 'pytest_timeout')

Ah, I probably got something wrong with the setuptool entrypoints or
something.  I'll look into it, thanks for finding this!


> I also played with the ability to use markers to override the default
> timeout at the test level (here is an excerpt of
> your code modified)
[...]

Thanks for this, I'll incorporate the timeout marker if you don't mind.

Regards,
Floris


-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org


From flub at devork.be  Fri Mar  9 00:08:20 2012
From: flub at devork.be (Floris Bruynooghe)
Date: Thu, 8 Mar 2012 23:08:20 +0000
Subject: [py-dev] New plugin: pytest-timeout
In-Reply-To: <20120307215549.GH26028@merlinux.eu>
References: <CAAWk_DxA3mh7GmNSRW5_EtGACH4qgeGQW2X4aZpGuiReP2Z6Ng@mail.gmail.com>
	<20120307215549.GH26028@merlinux.eu>
Message-ID: <CAAWk_Dy2UEYJf4ednPrCQ_+nvGus6zdX4dfY6QvZrf1MvwFKqA@mail.gmail.com>

On 7 March 2012 21:55, holger krekel <holger at merlinux.eu> wrote:
> looking at version 0.1 i wonder why you didn't go with always activating
> the plugin so not requiring any pytest_plugins setting. ?The default
> could be "no timeout" or None which could be modified with:
>
> ? ?[pytest]
> ? ?timeout = 2.3 # secs

This simply didn't occur to me, probably because personally I don't
install it but just have the module included in the root of our
project.  Thanks for the idea.

> or with a marker. ?Not using some implicit magic number is anyway a good
> idea i think.

Not sure what you mean.  Do you mean using 0 as saying "no timeout" is
a magic number but e.g. None is fine?  Essentially anything else other
then a positive number is "disable timeout" to me.

Regards,
Floris

-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org


From flub at devork.be  Fri Mar  9 00:17:19 2012
From: flub at devork.be (Floris Bruynooghe)
Date: Thu, 8 Mar 2012 23:17:19 +0000
Subject: [py-dev] New plugin: pytest-timeout
In-Reply-To: <4F57F0AC.7000603@gmx.de>
References: <CAAWk_DxA3mh7GmNSRW5_EtGACH4qgeGQW2X4aZpGuiReP2Z6Ng@mail.gmail.com>
	<20120307215549.GH26028@merlinux.eu> <4F57F0AC.7000603@gmx.de>
Message-ID: <CAAWk_DyS4eOu8bcPMUgJ7CrtufkQSf9n+uB55+DJ14fymt6TXQ@mail.gmail.com>

On 7 March 2012 23:35, Ronny Pfannschmidt <Ronny.Pfannschmidt at gmx.de> wrote:
> in addition something for eventlet/gevent based timeout objects
> as well as a ini option to choose the timeout method between signal/thread,
> gevent, eventlet could be nice

eventlet/gevent timeouts are also a good idea, I'll look into adding
these as well.

One issue with this however is that I hate calling import from inside
a function (partially because I've looked too close at eventlet's
patching code) but unconditionally importing it upfront, in a
try-except block, woud be bad for py.test's startup time.  Would
importing in a function be reasonable in this case?  Or am I missing
another solution?


Thanks for all the feedback everyone!

Floris


-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org


From holger at merlinux.eu  Fri Mar  9 00:32:11 2012
From: holger at merlinux.eu (holger krekel)
Date: Thu, 8 Mar 2012 23:32:11 +0000
Subject: [py-dev] New plugin: pytest-timeout
In-Reply-To: <CAAWk_Dy2UEYJf4ednPrCQ_+nvGus6zdX4dfY6QvZrf1MvwFKqA@mail.gmail.com>
References: <CAAWk_DxA3mh7GmNSRW5_EtGACH4qgeGQW2X4aZpGuiReP2Z6Ng@mail.gmail.com>
	<20120307215549.GH26028@merlinux.eu>
	<CAAWk_Dy2UEYJf4ednPrCQ_+nvGus6zdX4dfY6QvZrf1MvwFKqA@mail.gmail.com>
Message-ID: <20120308233211.GI26028@merlinux.eu>

On Thu, Mar 08, 2012 at 23:08 +0000, Floris Bruynooghe wrote:
> On 7 March 2012 21:55, holger krekel <holger at merlinux.eu> wrote:
> > or with a marker. ?Not using some implicit magic number is anyway a good
> > idea i think.
> 
> Not sure what you mean.  Do you mean using 0 as saying "no timeout" is
> a magic number but e.g. None is fine?  Essentially anything else other
> then a positive number is "disable timeout" to me.

just meant "5 seconds" as a default.  0 or None is fine to mean "no
timeout" IMHO.

best,
holger

> Regards,
> Floris
> 
> -- 
> Debian GNU/Linux -- The Power of Freedom
> www.debian.org | www.gnu.org | www.kernel.org
> 


From Ronny.Pfannschmidt at gmx.de  Fri Mar  9 00:42:31 2012
From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt)
Date: Fri, 09 Mar 2012 00:42:31 +0100
Subject: [py-dev] New plugin: pytest-timeout
In-Reply-To: <CAAWk_DyS4eOu8bcPMUgJ7CrtufkQSf9n+uB55+DJ14fymt6TXQ@mail.gmail.com>
References: <CAAWk_DxA3mh7GmNSRW5_EtGACH4qgeGQW2X4aZpGuiReP2Z6Ng@mail.gmail.com>
	<20120307215549.GH26028@merlinux.eu> <4F57F0AC.7000603@gmx.de>
	<CAAWk_DyS4eOu8bcPMUgJ7CrtufkQSf9n+uB55+DJ14fymt6TXQ@mail.gmail.com>
Message-ID: <4F5943E7.5080204@gmx.de>

On 03/09/2012 12:17 AM, Floris Bruynooghe wrote:
> On 7 March 2012 23:35, Ronny Pfannschmidt<Ronny.Pfannschmidt at gmx.de>  wrote:
>> in addition something for eventlet/gevent based timeout objects
>> as well as a ini option to choose the timeout method between signal/thread,
>> gevent, eventlet could be nice
>
> eventlet/gevent timeouts are also a good idea, I'll look into adding
> these as well.
>
> One issue with this however is that I hate calling import from inside
> a function (partially because I've looked too close at eventlet's
> patching code) but unconditionally importing it upfront, in a
> try-except block, woud be bad for py.test's startup time.  Would
> importing in a function be reasonable in this case?  Or am I missing
> another solution?
>

Lets go with the assumption that whoever is requesting a gevent/eventlet 
timeout knows enough of what hes doing
(since it would need a config entry)

then its reasonable enough to just grab the timeouts in the function

wrt monkeypatching, i think its strictly opt-in, and should also be left 
to the user

best,
Ronny

>
> Thanks for all the feedback everyone!
>
> Floris
>
>



From lpbrac at dolby.com  Fri Mar  9 01:28:14 2012
From: lpbrac at dolby.com (Brack, Laurent P.)
Date: Thu, 8 Mar 2012 16:28:14 -0800
Subject: [py-dev] New plugin: pytest-timeout
References: <CAAWk_DxA3mh7GmNSRW5_EtGACH4qgeGQW2X4aZpGuiReP2Z6Ng@mail.gmail.com><05BB196AB3DA6C4BBE11AB6C957581FE42A39DB3@sfo-exch-01.dolby.net><CAAWk_DyYqj=v7gvPdFx7mddx4BZbfqEYCDtD_Ofy1rhEuZ=euA@mail.gmail.com><05BB196AB3DA6C4BBE11AB6C957581FE42A39E96@sfo-exch-01.dolby.net><05BB196AB3DA6C4BBE11AB6C957581FE42A39E98@sfo-exch-01.dolby.net>
	<CAAWk_DxhuqMYV-Rp-B6mxSH3gKVFUcwdhNGEXgQKko4PXgjNJA@mail.gmail.com>
Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE42A39EDF@sfo-exch-01.dolby.net>

Please be my guest and use it as we are using your code. 

Thanks a lot for that plugin by the way. 


-----Original Message-----
From: floris.bruynooghe at gmail.com on behalf of Floris Bruynooghe
Sent: Thu 3/8/2012 2:49 PM
To: Brack, Laurent P.
Cc: Brard, Valentin; py-dev at codespeak.net
Subject: Re: [py-dev] New plugin: pytest-timeout
 
Hello Laurent,

On 7 March 2012 20:56, Brack, Laurent P. <lpbrac at dolby.com> wrote:
> Somehow (and I don't know why), registering the plugin as 'timeout' instead
> of 'pytest_timeout' was the cause of the problem,
> so for it to work on my side, I had to do:
>
> config.pluginmanager.register(FaultHandlerPlugin(config), 'pytest_timeout')

Ah, I probably got something wrong with the setuptool entrypoints or
something.  I'll look into it, thanks for finding this!


> I also played with the ability to use markers to override the default
> timeout at the test level (here is an excerpt of
> your code modified)
[...]

Thanks for this, I'll incorporate the timeout marker if you don't mind.

Regards,
Floris


-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20120308/564dce4f/attachment.html>

From flub at devork.be  Sat Mar 17 18:31:34 2012
From: flub at devork.be (Floris Bruynooghe)
Date: Sat, 17 Mar 2012 18:31:34 +0100
Subject: [py-dev] pytest-timeout 0.2
Message-ID: <CAAWk_DwoODtnt7XwfS5ngHzsw9bXscbApL+oPArs=4_gDbiR+A@mail.gmail.com>

Hi all,

I've made a second release of the pytest-timeout plugin for py.test
which can time out long running tests.  This release includes a number
of suggestions made on this list, major changes include:

* Fixed the activation problem
* Set timeout using configuration file
* Add a timeout marker to modify timeout of one item
* The marker can also choose the method (signal/thread)
* Renamed --nosigalrm to --timeout_method to future proof adding of
eventlet and gevent timeout methods
* Works on python 3, tested on 2.6, 2.7 and 3.2

Not yet done:

* Automatic enabling of the plugin, you still need to enable it on the
command line or configuration file before you can use the marker.
This was probably a bad idea but I felt bad about stealing a marker by
default.
* eventlet and gevent timeouts


As before the release is on pypi:
http://pypi.python.org/pypi/pytest-timeout and the development
repository and issue tracker on bitbucket:
https://bitbucket.org/flub/pytest-timeout/

I'd be pleased to receive any further feedback you may have.

Floris


-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org


From holger at merlinux.eu  Sun Mar 18 00:16:27 2012
From: holger at merlinux.eu (holger krekel)
Date: Sat, 17 Mar 2012 23:16:27 +0000
Subject: [py-dev] pytest-timeout 0.2
In-Reply-To: <CAAWk_DwoODtnt7XwfS5ngHzsw9bXscbApL+oPArs=4_gDbiR+A@mail.gmail.com>
References: <CAAWk_DwoODtnt7XwfS5ngHzsw9bXscbApL+oPArs=4_gDbiR+A@mail.gmail.com>
Message-ID: <20120317231627.GT26028@merlinux.eu>

Hi Floris,

On Sat, Mar 17, 2012 at 18:31 +0100, Floris Bruynooghe wrote:
> I've made a second release of the pytest-timeout plugin for py.test
> which can time out long running tests.  This release includes a number
> of suggestions made on this list, major changes include:
> 
> * Fixed the activation problem
> * Set timeout using configuration file
> * Add a timeout marker to modify timeout of one item
> * The marker can also choose the method (signal/thread)
> * Renamed --nosigalrm to --timeout_method to future proof adding of
> eventlet and gevent timeout methods
> * Works on python 3, tested on 2.6, 2.7 and 3.2
> 
> Not yet done:
> 
> * Automatic enabling of the plugin, you still need to enable it on the
> command line or configuration file before you can use the marker.
> This was probably a bad idea but I felt bad about stealing a marker by
> default.

I went ahead and created a test function with

    @pytest.mark.timeout(1)
    def test_hello():
        ...

but the timeout was not honoured.  Then i skimmed the docs :)
added "timeout_method = signal" to my ini-file and ran, still not honoured.
Then i figured i need to set some dummy "timeout = 10" in the ini - and now 
i get the proper timeout of 1 second.

I understand the hesitance to grab a general name like "timeout" but then
again installing "pytest-timeout" is a deliberate act and it grabbing the
"timeout" marker is not surprising IMO.  So i'd kindly encourage you to
go for it. I wonder btw. if the output of "--markers" should be merged with
"--help".  The latter would get yet longer but then again it's nice to 
have all the info at a fingertip.

Another feedback item: "@pytest.mark.timeout(5, 'signal')" ought to work.
It's slightly awkward because of the marker args/kwargs API but it's expected
from a pure user perspective i think.

Moreover i'd eventually like to include the timeout plugin
in pytest core.  It's an important feature for functional testing.

> * eventlet and gevent timeouts

Here is what i did for eventlet (only accessing the decorator here):

https://bitbucket.org/hpk42/detox/src/f9f8c0107cc1/tests/conftest.py#cl-108

cheers & thanks,
holger

> 
> As before the release is on pypi:
> http://pypi.python.org/pypi/pytest-timeout and the development
> repository and issue tracker on bitbucket:
> https://bitbucket.org/flub/pytest-timeout/
> 
> I'd be pleased to receive any further feedback you may have.
> 
> Floris
> 
> 
> -- 
> Debian GNU/Linux -- The Power of Freedom
> www.debian.org | www.gnu.org | www.kernel.org
> _______________________________________________
> py-dev mailing list
> py-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/py-dev
> 


From flub at devork.be  Mon Mar 19 00:24:07 2012
From: flub at devork.be (Floris Bruynooghe)
Date: Mon, 19 Mar 2012 00:24:07 +0100
Subject: [py-dev] pytest-timeout 0.2
In-Reply-To: <20120317231627.GT26028@merlinux.eu>
References: <CAAWk_DwoODtnt7XwfS5ngHzsw9bXscbApL+oPArs=4_gDbiR+A@mail.gmail.com>
	<20120317231627.GT26028@merlinux.eu>
Message-ID: <CAAWk_Dy3E=eee9qXv5P5qfOyVCF5KbyM3JAfVcTbqU=zP8UuhQ@mail.gmail.com>

Hello Holger,

On 18 March 2012 00:16, holger krekel <holger at merlinux.eu> wrote:
> I went ahead and created a test function with
>
> ? ?@pytest.mark.timeout(1)
> ? ?def test_hello():
> ? ? ? ?...
>
> but the timeout was not honoured. ?Then i skimmed the docs :)
> added "timeout_method = signal" to my ini-file and ran, still not honoured.
> Then i figured i need to set some dummy "timeout = 10" in the ini - and now
> i get the proper timeout of 1 second.

Actually timeout=0 would be fine too.

> I understand the hesitance to grab a general name like "timeout" but then
> again installing "pytest-timeout" is a deliberate act and it grabbing the
> "timeout" marker is not surprising IMO. ?So i'd kindly encourage you to
> go for it. I wonder btw. if the output of "--markers" should be merged with
> "--help". ?The latter would get yet longer but then again it's nice to
> have all the info at a fingertip.

In fact I had no idea --markers existed, I was considering brining
this up but I just hadn't looked around enough!  This does make me
feel a lot better about using the timeout marker and you've convinced
me it's outweighs the stumbling blocks of requiring --timeout=0.

> Another feedback item: "@pytest.mark.timeout(5, 'signal')" ought to work.
> It's slightly awkward because of the marker args/kwargs API but it's expected
> from a pure user perspective i think.

Yes, I did consider that but then python3 allows keyword only
arguments and this seemed a reasonable candidate while the api to make
it lends itself to it naturally.  But don't see any problem with
making it positional as well and am happy to add this.

> Moreover i'd eventually like to include the timeout plugin
> in pytest core. ?It's an important feature for functional testing.

That would be nice!  But as you implied it probably needs to mature a
bit more outside of pytest core first.


>> * eventlet and gevent timeouts
>
> Here is what i did for eventlet (only accessing the decorator here):
>
> https://bitbucket.org/hpk42/detox/src/f9f8c0107cc1/tests/conftest.py#cl-108

Thanks for that hint, hadn't started looking yet but this will save me
some experimenting.


Regards,
Floris


-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org


From tetsuya.morimoto at gmail.com  Mon Mar 19 04:23:33 2012
From: tetsuya.morimoto at gmail.com (Tetsuya Morimoto)
Date: Sun, 18 Mar 2012 20:23:33 -0700
Subject: [py-dev] pytest-quickcheck plugin
Message-ID: <CAGBzy2rf247o44xgQ7E+=r_KgfESR1ypPRyx=SK9aqtxXqOURw@mail.gmail.com>

Hi,

I've made my first pytest plugin named pytest-quickcheck for
generating random data.
http://pypi.python.org/pypi/pytest-quickcheck/

Originally, this idea is inspired by Haskell QuickCheck (and paycheck,
its Python implementation).
http://www.haskell.org/haskellwiki/Introduction_to_QuickCheck
http://pypi.python.org/pypi/paycheck/

I think it's useful to make sure of the robustness by generating lots
of test-data for data-driven test. However, this plugin is still
experimental beta, so I'll welcome any comments or suggestions on.

thanks,
Tetsuya


From lpbrac at dolby.com  Mon Mar 19 17:38:47 2012
From: lpbrac at dolby.com (Brack, Laurent P.)
Date: Mon, 19 Mar 2012 09:38:47 -0700
Subject: [py-dev] pytest-timeout 0.2
In-Reply-To: <CAAWk_Dy3E=eee9qXv5P5qfOyVCF5KbyM3JAfVcTbqU=zP8UuhQ@mail.gmail.com>
References: <CAAWk_DwoODtnt7XwfS5ngHzsw9bXscbApL+oPArs=4_gDbiR+A@mail.gmail.com><20120317231627.GT26028@merlinux.eu>
	<CAAWk_Dy3E=eee9qXv5P5qfOyVCF5KbyM3JAfVcTbqU=zP8UuhQ@mail.gmail.com>
Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE445528E2@sfo-exch-01.dolby.net>

To have this part of the core pytest would be great (even unit tests can hang).

It would also be nice to have a consistent behavior between sigalarm on/off. For instance, on Windows, pytest exits on first hang as opposed to *nix where the test is pre-empted and pytest moves on to the next one. 

Cheers

/Laurent

-----Original Message-----
From: py-dev-bounces at codespeak.net [mailto:py-dev-bounces at codespeak.net] On Behalf Of Floris Bruynooghe
Sent: Sunday, March 18, 2012 4:24 PM
To: holger krekel; Floris Bruynooghe; py-dev at codespeak.net
Subject: Re: [py-dev] pytest-timeout 0.2

Hello Holger,

On 18 March 2012 00:16, holger krekel <holger at merlinux.eu> wrote:
> I went ahead and created a test function with
>
> ? ?@pytest.mark.timeout(1)
> ? ?def test_hello():
> ? ? ? ?...
>
> but the timeout was not honoured. ?Then i skimmed the docs :) added 
> "timeout_method = signal" to my ini-file and ran, still not honoured.
> Then i figured i need to set some dummy "timeout = 10" in the ini - 
> and now i get the proper timeout of 1 second.

Actually timeout=0 would be fine too.

> I understand the hesitance to grab a general name like "timeout" but 
> then again installing "pytest-timeout" is a deliberate act and it 
> grabbing the "timeout" marker is not surprising IMO. ?So i'd kindly 
> encourage you to go for it. I wonder btw. if the output of "--markers" 
> should be merged with "--help". ?The latter would get yet longer but 
> then again it's nice to have all the info at a fingertip.

In fact I had no idea --markers existed, I was considering brining this up but I just hadn't looked around enough!  This does make me feel a lot better about using the timeout marker and you've convinced me it's outweighs the stumbling blocks of requiring --timeout=0.

> Another feedback item: "@pytest.mark.timeout(5, 'signal')" ought to work.
> It's slightly awkward because of the marker args/kwargs API but it's 
> expected from a pure user perspective i think.

Yes, I did consider that but then python3 allows keyword only arguments and this seemed a reasonable candidate while the api to make it lends itself to it naturally.  But don't see any problem with making it positional as well and am happy to add this.

> Moreover i'd eventually like to include the timeout plugin in pytest 
> core. ?It's an important feature for functional testing.

That would be nice!  But as you implied it probably needs to mature a bit more outside of pytest core first.


>> * eventlet and gevent timeouts
>
> Here is what i did for eventlet (only accessing the decorator here):
>
> https://bitbucket.org/hpk42/detox/src/f9f8c0107cc1/tests/conftest.py#c
> l-108

Thanks for that hint, hadn't started looking yet but this will save me some experimenting.


Regards,
Floris


--
Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org _______________________________________________
py-dev mailing list
py-dev at codespeak.net
http://codespeak.net/mailman/listinfo/py-dev

From flub at devork.be  Mon Mar 19 23:20:56 2012
From: flub at devork.be (Floris Bruynooghe)
Date: Mon, 19 Mar 2012 22:20:56 +0000
Subject: [py-dev] pytest-timeout 0.2
In-Reply-To: <05BB196AB3DA6C4BBE11AB6C957581FE445528E2@sfo-exch-01.dolby.net>
References: <CAAWk_DwoODtnt7XwfS5ngHzsw9bXscbApL+oPArs=4_gDbiR+A@mail.gmail.com>
	<20120317231627.GT26028@merlinux.eu>
	<CAAWk_Dy3E=eee9qXv5P5qfOyVCF5KbyM3JAfVcTbqU=zP8UuhQ@mail.gmail.com>
	<05BB196AB3DA6C4BBE11AB6C957581FE445528E2@sfo-exch-01.dolby.net>
Message-ID: <CAAWk_Dw2XTQK8P7vDEFK7xRkWsZiNm1dT1Pp2OCZEPAsFaQBrg@mail.gmail.com>

On 19 March 2012 16:38, Brack, Laurent P. <lpbrac at dolby.com> wrote:
> It would also be nice to have a consistent behavior between sigalarm on/off. For instance, on Windows, pytest exits on first hang as opposed to *nix where the test is pre-empted and pytest moves on to the next one.

I was writing up a response saying I didn't know how to do so, but
while doing so I thought it might be possible to emulate sigalrm with
a timer thread which fires an other signal.  I might investigate the
feasibility of such an approach.  I do worry about gratuitously using
signals however, the simple timer thread with os._exit() may always
have to stay as the fail-safe option.

If anyone has any other ideas about this please let me know.

Regards,
Floris


-- 
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org


From lpbrac at dolby.com  Tue Mar 20 00:47:36 2012
From: lpbrac at dolby.com (Brack, Laurent P.)
Date: Mon, 19 Mar 2012 16:47:36 -0700
Subject: [py-dev] pytest-timeout 0.2
In-Reply-To: <CAAWk_Dw2XTQK8P7vDEFK7xRkWsZiNm1dT1Pp2OCZEPAsFaQBrg@mail.gmail.com>
References: <CAAWk_DwoODtnt7XwfS5ngHzsw9bXscbApL+oPArs=4_gDbiR+A@mail.gmail.com><20120317231627.GT26028@merlinux.eu><CAAWk_Dy3E=eee9qXv5P5qfOyVCF5KbyM3JAfVcTbqU=zP8UuhQ@mail.gmail.com><05BB196AB3DA6C4BBE11AB6C957581FE445528E2@sfo-exch-01.dolby.net>
	<CAAWk_Dw2XTQK8P7vDEFK7xRkWsZiNm1dT1Pp2OCZEPAsFaQBrg@mail.gmail.com>
Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE445B4106@sfo-exch-01.dolby.net>

I don't want to make this issue overly complicated but here are some scenarios (that we have).

Our tests, in some instances, interact with python extensions written in C or C++ (or in some case COM inproc server - yuk) which can be multi-threaded.
 
If the hang occurs in a critical section in the C/C++ domain I am not sure what the timeout effect would be but 
Assuming that the test is pre-empted in a critical section (of the extension), we would most likely be doomed anyway (mutexes would not be released). Next test would hang, be aborted and so on. 

In our case, the safest thing to do is to run with xdist -n 1 so then the OS takes care of doing most if not all the cleanup.
I am not so intimate with xdist but I was hoping that upon detecting that the "remote process" had died, 
xdist would restart a new one. 

In this case pytest ended with:

[gw0] win32 -- Python 2.7.2 C:\Python27\python.exe
Slave 'gw0' crashed while running 'src/timeout/test_sample.py::TestTimeOut::()::test_hang'

Which signaled the end of everything (other tests were not run). This is a something we will have to solve internally 
(and when we have the solution will be more than happy to contribute it back). We might be able to restart a remote 
Process on pytest_testnodedown(). We would be however too late to run any restore() and chances are that those might hang as well.

More food for thoughts than anything at this point. 

Floris, thanks for your work. We are not yet in a position where we can help much but we will get to it.

/Laurent

P.S.: Speaking of xdist, further down the road we are interested in extending xdist (via its hooks) to interface it with OpenStack (dynamic provisioning of VM). Anyone has any use for this or thinking about the same thing?

-----Original Message-----
From: floris.bruynooghe at gmail.com [mailto:floris.bruynooghe at gmail.com] On Behalf Of Floris Bruynooghe
Sent: Monday, March 19, 2012 3:21 PM
To: Brack, Laurent P.
Cc: holger krekel; py-dev at codespeak.net; Brard, Valentin
Subject: Re: [py-dev] pytest-timeout 0.2

On 19 March 2012 16:38, Brack, Laurent P. <lpbrac at dolby.com> wrote:
> It would also be nice to have a consistent behavior between sigalarm on/off. For instance, on Windows, pytest exits on first hang as opposed to *nix where the test is pre-empted and pytest moves on to the next one.

I was writing up a response saying I didn't know how to do so, but while doing so I thought it might be possible to emulate sigalrm with a timer thread which fires an other signal.  I might investigate the feasibility of such an approach.  I do worry about gratuitously using signals however, the simple timer thread with os._exit() may always have to stay as the fail-safe option.

If anyone has any other ideas about this please let me know.

Regards,
Floris


--
Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org

From holger at merlinux.eu  Tue Mar 20 07:10:12 2012
From: holger at merlinux.eu (holger krekel)
Date: Tue, 20 Mar 2012 06:10:12 +0000
Subject: [py-dev] pytest-timeout 0.2
In-Reply-To: <05BB196AB3DA6C4BBE11AB6C957581FE445B4106@sfo-exch-01.dolby.net>
References: <CAAWk_DwoODtnt7XwfS5ngHzsw9bXscbApL+oPArs=4_gDbiR+A@mail.gmail.com>
	<20120317231627.GT26028@merlinux.eu>
	<CAAWk_Dy3E=eee9qXv5P5qfOyVCF5KbyM3JAfVcTbqU=zP8UuhQ@mail.gmail.com>
	<05BB196AB3DA6C4BBE11AB6C957581FE445528E2@sfo-exch-01.dolby.net>
	<CAAWk_Dw2XTQK8P7vDEFK7xRkWsZiNm1dT1Pp2OCZEPAsFaQBrg@mail.gmail.com>
	<05BB196AB3DA6C4BBE11AB6C957581FE445B4106@sfo-exch-01.dolby.net>
Message-ID: <20120320061012.GC26028@merlinux.eu>

Hi Laurent, Floris,

On Mon, Mar 19, 2012 at 16:47 -0700, Brack, Laurent P. wrote:
> I don't want to make this issue overly complicated but here are some scenarios (that we have).

proper termination and handling of hanging tests is involved.  So 
discussion and hearing use cases makes sense to me.

> Our tests, in some instances, interact with python extensions written in C or C++ (or in some case COM inproc server - yuk) which can be multi-threaded.
>  
> If the hang occurs in a critical section in the C/C++ domain I am not sure what the timeout effect would be but 
> Assuming that the test is pre-empted in a critical section (of the extension), we would most likely be doomed anyway (mutexes would not be released). Next test would hang, be aborted and so on. 
> 
> In our case, the safest thing to do is to run with xdist -n 1 so then the OS takes care of doing most if not all the cleanup.
> I am not so intimate with xdist but I was hoping that upon detecting that the "remote process" had died, 
> xdist would restart a new one. 

It's reasonable to expect this from the xdist plugin, even when you run with
"-n N" with N>1.

> In this case pytest ended with:
> 
> [gw0] win32 -- Python 2.7.2 C:\Python27\python.exe
> Slave 'gw0' crashed while running 'src/timeout/test_sample.py::TestTimeOut::()::test_hang'
> 
> Which signaled the end of everything (other tests were not run). This is a something we will have to solve internally 
> (and when we have the solution will be more than happy to contribute it back). We might be able to restart a remote 
> Process on pytest_testnodedown(). We would be however too late to run any restore() and chances are that those might hang as well.

Indeed, no attempt is made to restart a node properly. It remains offline
and when run with "-n1" everything dies because no node is left to continue.
Not sure how hard it is to try to fix it.

> More food for thoughts than anything at this point. 
> 
> Floris, thanks for your work. We are not yet in a position where we can help much but we will get to it.

Coming to think of it, the xdist plugin might also do "timeout" handling
and send a keyboard interrupt from the outside to a running node.

> /Laurent
> 
> P.S.: Speaking of xdist, further down the road we are interested in extending xdist (via its hooks) to interface it with OpenStack (dynamic provisioning of VM). Anyone has any use for this or thinking about the same thing?

I am interested in OpenStack but you can you detail a bit more of what
you want to achieve?

best,
holger

> 
> -----Original Message-----
> From: floris.bruynooghe at gmail.com [mailto:floris.bruynooghe at gmail.com] On Behalf Of Floris Bruynooghe
> Sent: Monday, March 19, 2012 3:21 PM
> To: Brack, Laurent P.
> Cc: holger krekel; py-dev at codespeak.net; Brard, Valentin
> Subject: Re: [py-dev] pytest-timeout 0.2
> 
> On 19 March 2012 16:38, Brack, Laurent P. <lpbrac at dolby.com> wrote:
> > It would also be nice to have a consistent behavior between sigalarm on/off. For instance, on Windows, pytest exits on first hang as opposed to *nix where the test is pre-empted and pytest moves on to the next one.
> 
> I was writing up a response saying I didn't know how to do so, but while doing so I thought it might be possible to emulate sigalrm with a timer thread which fires an other signal.  I might investigate the feasibility of such an approach.  I do worry about gratuitously using signals however, the simple timer thread with os._exit() may always have to stay as the fail-safe option.
> 
> If anyone has any other ideas about this please let me know.
> 
> Regards,
> Floris
> 
> 
> --
> Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org


From lpbrac at dolby.com  Tue Mar 20 17:43:46 2012
From: lpbrac at dolby.com (Brack, Laurent P.)
Date: Tue, 20 Mar 2012 09:43:46 -0700
Subject: [py-dev] pytest-timeout 0.2
References: <CAAWk_DwoODtnt7XwfS5ngHzsw9bXscbApL+oPArs=4_gDbiR+A@mail.gmail.com>
	<20120317231627.GT26028@merlinux.eu>
	<CAAWk_Dy3E=eee9qXv5P5qfOyVCF5KbyM3JAfVcTbqU=zP8UuhQ@mail.gmail.com>
	<05BB196AB3DA6C4BBE11AB6C957581FE445528E2@sfo-exch-01.dolby.net>
	<CAAWk_Dw2XTQK8P7vDEFK7xRkWsZiNm1dT1Pp2OCZEPAsFaQBrg@mail.gmail.com>
	<05BB196AB3DA6C4BBE11AB6C957581FE445B4106@sfo-exch-01.dolby.net>
	<20120320061012.GC26028@merlinux.eu>
Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE42A3A081@sfo-exch-01.dolby.net>

Hi Holger,

I am not saying there is anything wrong with the way xdist is handling a node going down 
at this point. the -n 1 was just my attempt to see if the timeout plugin would behave 
differently when running the test out of process. 

I think it did, as pytest did not exit the way it does without xdist ... it just ran out of nodes
this time and could not carry forward. 

> Coming to think of it, the xdist plugin might also do "timeout" handling
> and send a keyboard interrupt from the outside to a running node.

I think this is a good way of handling this. As Floris pointed out, there is always a danger
of using signal (right now his marker allows to switch modes on a per test basis which is 
a good thing). 

> I am interested in OpenStack but you can you detail a bit more of what
> you want to achieve?

I will start a separate thread - OpenStack (as it is somewhat unrelated to the timeout subject).

/Laurent

-----Original Message-----
From: holger krekel [mailto:holger at merlinux.eu]
Sent: Mon 3/19/2012 11:10 PM
To: Brack, Laurent P.
Cc: Floris Bruynooghe; holger krekel; py-dev at codespeak.net; Brard, Valentin
Subject: Re: [py-dev] pytest-timeout 0.2
 
Hi Laurent, Floris,

On Mon, Mar 19, 2012 at 16:47 -0700, Brack, Laurent P. wrote:
> I don't want to make this issue overly complicated but here are some scenarios (that we have).

proper termination and handling of hanging tests is involved.  So 
discussion and hearing use cases makes sense to me.

> Our tests, in some instances, interact with python extensions written in C or C++ (or in some case COM inproc server - yuk) which can be multi-threaded.
>  
> If the hang occurs in a critical section in the C/C++ domain I am not sure what the timeout effect would be but 
> Assuming that the test is pre-empted in a critical section (of the extension), we would most likely be doomed anyway (mutexes would not be released). Next test would hang, be aborted and so on. 
> 
> In our case, the safest thing to do is to run with xdist -n 1 so then the OS takes care of doing most if not all the cleanup.
> I am not so intimate with xdist but I was hoping that upon detecting that the "remote process" had died, 
> xdist would restart a new one. 

It's reasonable to expect this from the xdist plugin, even when you run with
"-n N" with N>1.

> In this case pytest ended with:
> 
> [gw0] win32 -- Python 2.7.2 C:\Python27\python.exe
> Slave 'gw0' crashed while running 'src/timeout/test_sample.py::TestTimeOut::()::test_hang'
> 
> Which signaled the end of everything (other tests were not run). This is a something we will have to solve internally 
> (and when we have the solution will be more than happy to contribute it back). We might be able to restart a remote 
> Process on pytest_testnodedown(). We would be however too late to run any restore() and chances are that those might hang as well.

Indeed, no attempt is made to restart a node properly. It remains offline
and when run with "-n1" everything dies because no node is left to continue.
Not sure how hard it is to try to fix it.

> More food for thoughts than anything at this point. 
> 
> Floris, thanks for your work. We are not yet in a position where we can help much but we will get to it.

Coming to think of it, the xdist plugin might also do "timeout" handling
and send a keyboard interrupt from the outside to a running node.

> /Laurent
> 
> P.S.: Speaking of xdist, further down the road we are interested in extending xdist (via its hooks) to interface it with OpenStack (dynamic provisioning of VM). Anyone has any use for this or thinking about the same thing?

I am interested in OpenStack but you can you detail a bit more of what
you want to achieve?

best,
holger

> 
> -----Original Message-----
> From: floris.bruynooghe at gmail.com [mailto:floris.bruynooghe at gmail.com] On Behalf Of Floris Bruynooghe
> Sent: Monday, March 19, 2012 3:21 PM
> To: Brack, Laurent P.
> Cc: holger krekel; py-dev at codespeak.net; Brard, Valentin
> Subject: Re: [py-dev] pytest-timeout 0.2
> 
> On 19 March 2012 16:38, Brack, Laurent P. <lpbrac at dolby.com> wrote:
> > It would also be nice to have a consistent behavior between sigalarm on/off. For instance, on Windows, pytest exits on first hang as opposed to *nix where the test is pre-empted and pytest moves on to the next one.
> 
> I was writing up a response saying I didn't know how to do so, but while doing so I thought it might be possible to emulate sigalrm with a timer thread which fires an other signal.  I might investigate the feasibility of such an approach.  I do worry about gratuitously using signals however, the simple timer thread with os._exit() may always have to stay as the fail-safe option.
> 
> If anyone has any other ideas about this please let me know.
> 
> Regards,
> Floris
> 
> 
> --
> Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20120320/37a501a7/attachment.html>

From lpbrac at dolby.com  Tue Mar 20 18:27:00 2012
From: lpbrac at dolby.com (Brack, Laurent P.)
Date: Tue, 20 Mar 2012 10:27:00 -0700
Subject: [py-dev] OpenStack
Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE42A3A082@sfo-exch-01.dolby.net>

Forking from the [py-dev] pytest-timeout 0.2 e-mail thread.

> I am interested in OpenStack but you can you detail a bit more of what
> you want to achieve?

I have attached a rough diagram of what we are building internally (hopefully it will not be filtered out). 

About a year ago, we attended a presentation on OpenStack (when it was still driven by Anso Labs before they got acquired by Rackspace). 
We were (and are) currently using a private cloud (VMWare) but contemplated the idea of scaling up to public clouds. Problem we had was that 
we had to develop custom code for each cloud vendor whether Amazon EC2, Penguin, VMWare, etc. so OpenStack was really a viable path to not lock 
ourselves in with a given vendor. 

While a bulk of our tests require embedded devices (in which case virtualization makes little sense) a good chunk can be run 
on standard workstations (all OS flavors) and in this case it only makes sense to move to a virtual environment. 

PyTest combined with xdist was the dream come true as we could focus on developing tests meant to run on 
a single machine and later seamlessly parallelize their execution. There is nothing new here. 

We were thinking of writing a plugin to xdist (cxdist - c for cloud) that would interface with OpenStack, using the pytest_xdist_setupnodes
and pytest_configure_node hooks.

In those hooks, the plugin would provision the machine (via OpenStack) and then make xdist believe that it is dealing with 
physical slaves.  

Finally at the end of the run, we would teardown the slaves, leaving resources for other tests to run. 

We have not started yet on this as scalability is not an issue but this is our plan. As the diagram shows, the read boxes 
are plugins we intend to release to the open source community. 

Our teams write a lot of "data driven tests" (testing various AV codecs with different configuration parameters using very similar verification procedure)
and as a result we make heavy use of funcargs. 

The pytest_testlink plugin is very similar to the pytest Case Conductor plugin from the Mozilla folks (http://blargon7.com/2012/01/case-conductor-pytest-plugin-proposal/) but interacts with TestLink (teamst.org in which we are currently involved in - improving performance of web services etc.) as opposed to Case Conductor. 

However in addition to linking a test case ID to a given python test, we have implemented a simple mechanism to generate test cases 
from what we called Test Description Records. Those are currently stored in excel spreadsheets but will be store in TestLink as 
test case properties. The object injection is done automatically by the plugin using a custom marker indicating which function argument shall 
be used for TDR injection as well as correlating TDR with parametrized test functions. 

The point is that by using the funcarg feature offered by pytest, we can generate tremendous amount of test cases with little effort
and this will require scalability and use of virtualization over time. 

Sorry for the long description, but I wanted to give a complete overview to help comprehend where we are going. 
Our plan is to eventually release all this to the open source community (when we think the plugins are mature and robust enough). 

If other people have a need/interest for those plugins (some of them are not illustrated), I would be more than happy to accelerate 
this process (getting approval from our Open Source Board). Since pytest_cxdist doesn't exist, this could be open source from the 
get go. 

Best 

Laurent
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20120320/b99c4f2a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: GTI Plugin.png
Type: image/png
Size: 59280 bytes
Desc: GTI Plugin.png
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20120320/b99c4f2a/attachment.png>

From holger at merlinux.eu  Sat Mar 24 14:41:58 2012
From: holger at merlinux.eu (holger krekel)
Date: Sat, 24 Mar 2012 13:41:58 +0000
Subject: [py-dev] OpenStack
In-Reply-To: <05BB196AB3DA6C4BBE11AB6C957581FE42A3A082@sfo-exch-01.dolby.net>
References: <05BB196AB3DA6C4BBE11AB6C957581FE42A3A082@sfo-exch-01.dolby.net>
Message-ID: <20120324134157.GM26028@merlinux.eu>

Hi Brack,

sorry for taking a while.  Am still on travel, currently in the Sonoran
deserts.  My answers up until first half May are thus likely to lag.
(They do have suprisingly good internet connection ... better than in the
three-times more expensive hotel in the San Francisco valley last week).

On Tue, Mar 20, 2012 at 10:27 -0700, Brack, Laurent P. wrote:
> Forking from the [py-dev] pytest-timeout 0.2 e-mail thread.
> 
> > I am interested in OpenStack but you can you detail a bit more of what
> > you want to achieve?
> 
> I have attached a rough diagram of what we are building internally (hopefully it will not be filtered out). 

thanks for sharing.

> About a year ago, we attended a presentation on OpenStack (when it was still driven by Anso Labs before they got acquired by Rackspace). 
> We were (and are) currently using a private cloud (VMWare) but contemplated the idea of scaling up to public clouds. Problem we had was that 
> we had to develop custom code for each cloud vendor whether Amazon EC2, Penguin, VMWare, etc. so OpenStack was really a viable path to not lock 
> ourselves in with a given vendor. 

sounds sensible.

> While a bulk of our tests require embedded devices (in which case virtualization makes little sense) a good chunk can be run 
> on standard workstations (all OS flavors) and in this case it only makes sense to move to a virtual environment. 
> 
> PyTest combined with xdist was the dream come true as we could focus on developing tests meant to run on 
> a single machine and later seamlessly parallelize their execution. There is nothing new here. 
> 
> We were thinking of writing a plugin to xdist (cxdist - c for cloud) that would interface with OpenStack, using the pytest_xdist_setupnodes
> and pytest_configure_node hooks.

ok.

> In those hooks, the plugin would provision the machine (via OpenStack) and then make xdist believe that it is dealing with 
> physical slaves.  

> Finally at the end of the run, we would teardown the slaves, leaving resources for other tests to run. 
> 
> We have not started yet on this as scalability is not an issue but this is our plan. As the diagram shows, the read boxes 
> are plugins we intend to release to the open source community. 

Is your general idea to use the open stack integration to get 
parallelizable test runs with totally controllable environments? 

> Our teams write a lot of "data driven tests" (testing various AV codecs with different configuration parameters using very similar verification procedure)
> and as a result we make heavy use of funcargs. 

Curious, are you using 2.2.3 for this?  There are some plans to further
improve parametrization where i would be interested in your feedback.

> The pytest_testlink plugin is very similar to the pytest Case Conductor plugin from the Mozilla folks (http://blargon7.com/2012/01/case-conductor-pytest-plugin-proposal/) but interacts with TestLink (teamst.org in which we are currently involved in - improving performance of web services etc.) as opposed to Case Conductor. 

Is pytest-testlink a plugin that you plan to work on?

> However in addition to linking a test case ID to a given python test, we have implemented a simple mechanism to generate test cases 
> from what we called Test Description Records. Those are currently stored in excel spreadsheets but will be store in TestLink as 
> test case properties. The object injection is done automatically by the plugin using a custom marker indicating which function argument shall 
> be used for TDR injection as well as correlating TDR with parametrized test functions. 

right.  Driving things from a spreadsheet, maybe even in Google docs, 
sounds good to me, btw :)

> The point is that by using the funcarg feature offered by pytest, we can generate tremendous amount of test cases with little effort
> and this will require scalability and use of virtualization over time. 
> 
> Sorry for the long description, but I wanted to give a complete overview to help comprehend where we are going. 
> Our plan is to eventually release all this to the open source community (when we think the plugins are mature and robust enough). 
> 
> If other people have a need/interest for those plugins (some of them are not illustrated), I would be more than happy to accelerate 
> this process (getting approval from our Open Source Board). Since pytest_cxdist doesn't exist, this could be open source from the 
> get go. 

I wonder if the provisioning of VMs wouldn't better fit at the level of
"tox", see http://tox.testrun.org or even yet a level higher.  tox
creates virtual environments on the Python level.  FYI there are plans to
introduce a plugin system to tox and to make pytest-xdist make use of
tox configuration directly.   

Btw, from May on i (and others) should be available through my company
merlinux for some consulting in case you need some more substantial
support than occassional comments from far away deserts :)

best,
holger


From lpbrac at dolby.com  Tue Mar 27 18:23:08 2012
From: lpbrac at dolby.com (Brack, Laurent P.)
Date: Tue, 27 Mar 2012 09:23:08 -0700
Subject: [py-dev] OpenStack
In-Reply-To: <20120324134157.GM26028@merlinux.eu>
References: <05BB196AB3DA6C4BBE11AB6C957581FE42A3A082@sfo-exch-01.dolby.net>
	<20120324134157.GM26028@merlinux.eu>
Message-ID: <05BB196AB3DA6C4BBE11AB6C957581FE449EF695@sfo-exch-01.dolby.net>

Hi Holger,

No problem for the delayed response. I know you were in the bay area a
while back and may have felt everyone is always in a rush but this is
not our case :)

-----Original Message-----
From: holger krekel [mailto:holger at merlinux.eu] 
Sent: Saturday, March 24, 2012 6:42 AM
To: Brack, Laurent P.
Cc: py-dev at codespeak.net
Subject: Re: [py-dev] OpenStack

Hi Brack,
[Laurent] >> This is my last name :)

sorry for taking a while.  Am still on travel, currently in the Sonoran
deserts.  My answers up until first half May are thus likely to lag.
(They do have suprisingly good internet connection ... better than in
the three-times more expensive hotel in the San Francisco valley last
week).

On Tue, Mar 20, 2012 at 10:27 -0700, Brack, Laurent P. wrote:
> Forking from the [py-dev] pytest-timeout 0.2 e-mail thread.
> 
> > I am interested in OpenStack but you can you detail a bit more of 
> > what you want to achieve?
> 
> I have attached a rough diagram of what we are building internally
(hopefully it will not be filtered out). 

thanks for sharing.

> About a year ago, we attended a presentation on OpenStack (when it was
still driven by Anso Labs before they got acquired by Rackspace). 
> We were (and are) currently using a private cloud (VMWare) but 
> contemplated the idea of scaling up to public clouds. Problem we had 
> was that we had to develop custom code for each cloud vendor whether
Amazon EC2, Penguin, VMWare, etc. so OpenStack was really a viable path
to not lock ourselves in with a given vendor.

sounds sensible.

> While a bulk of our tests require embedded devices (in which case 
> virtualization makes little sense) a good chunk can be run on standard
workstations (all OS flavors) and in this case it only makes sense to
move to a virtual environment.
> 
> PyTest combined with xdist was the dream come true as we could focus 
> on developing tests meant to run on a single machine and later
seamlessly parallelize their execution. There is nothing new here.
> 
> We were thinking of writing a plugin to xdist (cxdist - c for cloud) 
> that would interface with OpenStack, using the pytest_xdist_setupnodes
and pytest_configure_node hooks.

ok.

> In those hooks, the plugin would provision the machine (via OpenStack)

> and then make xdist believe that it is dealing with physical slaves.

> Finally at the end of the run, we would teardown the slaves, leaving
resources for other tests to run. 
> 
> We have not started yet on this as scalability is not an issue but 
> this is our plan. As the diagram shows, the read boxes are plugins we
intend to release to the open source community.

Is your general idea to use the open stack integration to get
parallelizable test runs with totally controllable environments? 

[Laurent] >> It is. We have a fairly large VMWare infrastructure but I
feel we are locked in. Also there is a point where scaling up this
private cloud will simply make no sense from an economic standpoint.
Finally, infrastructure like VMWare are of little use for the open
source community. 

> Our teams write a lot of "data driven tests" (testing various AV 
> codecs with different configuration parameters using very similar
verification procedure) and as a result we make heavy use of funcargs.

Curious, are you using 2.2.3 for this?  There are some plans to further
improve parametrization where i would be interested in your feedback.

[Laurent] >> Yes. As a matter of fact we had to make changes to our
plugin to work properly with the new metafunc.parametrize method. Maybe
here is the time to open a small parenthesis. We have simplified the
generation process
For our common users, although this doesn't prevent them from using
custom hook implementations or factories. One thing that the plugin does
is attach "meta data" to items (which is carried over from hook to
hook). This meta data contains information about test cases on the
testlink Server. One of the challenge with metafunc was to find a way to
attach this information in a way it would not be lost during the test
generation done by pytest. In 2.1.3, with "addcall" we had done this by
hacking  the "param" formal parameter intended use (while preserving the
functionality). In 2.2.3 (the addcall hack was broken - meta data got
lost), we found another way. Again a hack and therefore no guaranty it
will work with future versions. It would be nice to have a supported way
to "attach" data as part of the parametrize process which is then
carried over to the "item" being generated. The idea is to carry over
information that may be generated by a hook to another hook (whenever it
makes sense).

> The pytest_testlink plugin is very similar to the pytest Case
Conductor plugin from the Mozilla folks
(http://blargon7.com/2012/01/case-conductor-pytest-plugin-proposal/) but
interacts with TestLink (teamst.org in which we are currently involved
in - improving performance of web services etc.) as opposed to Case
Conductor. 

Is pytest-testlink a plugin that you plan to work on?
[Laurent] >> Actually we have it working. We have been using it for
about 6 months now and I just made a new internal release (to fix the
issue with 2.1.3+) which also supports "platforms" (a feature of
TestLink) and filtering based on TestLink constructs (outcome, test case
ID, keywords). The plugin relies on another package (called pytl) which
provides an OO model of the testlink server built on top of their
existing web services. Those web services are quite buggy and incomplete
but we have built-in workarounds. In parallel we are working with the
TestLink team to provide a more complete set of web services. We are
also working on restructuring the code base allowing better integration
via plugins (requirement managements, reporting - we have a solution
using BIRT) as well performance improvements, test case libraries (for
re-use between projects), project groups, etc. While the pytest testlink
plugin is extensively documented and so far proven to be pretty robust,
I want to clean it up even more before releasing it (making it available
as an open source package was the intent from the start).

> However in addition to linking a test case ID to a given python test, 
> we have implemented a simple mechanism to generate test cases from 
> what we called Test Description Records. Those are currently stored in

> excel spreadsheets but will be store in TestLink as test case
properties. The object injection is done automatically by the plugin
using a custom marker indicating which function argument shall be used
for TDR injection as well as correlating TDR with parametrized test
functions.

right.  Driving things from a spreadsheet, maybe even in Google docs,
sounds good to me, btw :)
[Laurent] >> We considered this but decided otherwise just because of
the limited functionality offered compared to M$ Office or LibreOffice
(we are actually using xlrd/xlwr). As a matter of fact I wanted to find
a way to abstract the "data source" for the TDR. Google doc would be a
perfect exercise.

> The point is that by using the funcarg feature offered by pytest, we 
> can generate tremendous amount of test cases with little effort and
this will require scalability and use of virtualization over time.
> 
> Sorry for the long description, but I wanted to give a complete
overview to help comprehend where we are going. 
> Our plan is to eventually release all this to the open source
community (when we think the plugins are mature and robust enough). 
> 
> If other people have a need/interest for those plugins (some of them 
> are not illustrated), I would be more than happy to accelerate this 
> process (getting approval from our Open Source Board). Since
pytest_cxdist doesn't exist, this could be open source from the get go.

I wonder if the provisioning of VMs wouldn't better fit at the level of
"tox", see http://tox.testrun.org or even yet a level higher.  tox
creates virtual environments on the Python level.  FYI there are plans
to introduce a plugin system to tox and to make pytest-xdist make use of
tox configuration directly.   
[Laurent] >> I need to educate myself more on Tox. We are using tox for
our CI system (Jenkins) but environment setup is actually done via
buildout (tox invoking buildout - maybe we would have worked with tox
from the start if we had known about it).
But the tox idea is pretty good as it this is really where the testbed
should be setup rather than runtime (pytest) when things can go wrong. 

Btw, from May on i (and others) should be available through my company
merlinux for some consulting in case you need some more substantial
support than occassional comments from far away deserts :)
[Laurent] >> Actually this sounds like interesting. Let me think about
it. We are a rather small team so we can use all the help we can get.
Enjoy and drink a lot of water :)

Cheers/Laurent
best,
holger


From mantihor at gmail.com  Sat Mar 31 17:16:55 2012
From: mantihor at gmail.com (Bogdan Opanchuk)
Date: Sun, 1 Apr 2012 02:16:55 +1100
Subject: [py-dev] Performance tests with py.test
Message-ID: <CAFHN2SOkDfn+nQ-30JydGKOWwWC8VwFxB9wWURVu=aFqTLtNRw@mail.gmail.com>

Hello,

I am trying to wrap performance tests for my module into py.test. Each
testcase is, basically, a data preparation stage (can take some time)
and the test itself. Is there some way (hooks, plugins) to make
py.test print the value returned from the testcase (and make testcases
time whatever they want)? What I want is just a table like the one
after "py.test -v", but with return values instead of "PASSED". I
googled it and haven't found any solutions.

At the moment I am doing it as following:
1. Overload "pytest_pyfunc_call()" and save returned result into
"pyfuncitem" parameter.
2. Overload "pytest_runtest_makereport()", look for "perf" mark and
somehow output "item.result".
3. I haven't decided about the way to output it yet, because I am
currently trying to find the place where "PASSED" is printed.
But this seems quite clunky and overcomplicated. Is there any better
way to do this? If not for the data preparation, the currently
existing time measuring system would probably be enough for me
(although I'd still miss the ability to print GFLOPS value instead of
seconds), but the data preparation is testcase-specific and I do not
want to move it to "pytest_generate_tests()" hook.

Thank you in advance.