Mailman 3 November 2006 - pytest-dev

[py-dev] Failing to manage test state of a module with py.test
by Pierre Rouleau Jan. 15, 2007

Jan. 15, 2007

Hi all, Using the latest of py.test (svn 34683), I can't seem to be able to control a module test setup using setup_module() as described in section 2.15 of http://codespeak.net/py/current/doc/test.html The document gives the following example: def setup_module(module): """ setup up any state specific to the execution of the given module. """ Now, I am assuming that the the definition of setup_module() must be written /inside/ the test script that test the module-under-test tested, right? Second, I am also assuming that setup_module's argument is the name of the module-under-test. Given the above assumptions, if I write a test script to test roman.py would be called test_roman.py and would include def setup_module(roman): print 'SETTING up roman for testing....' And if I ran py.test -s test_roman.py I should be able to see the printed output. I don't. So, what do i do wrong here? Thanks

4 6

[py-dev] Another very small typo in test/defaultconftest.py
by Pierre Rouleau Nov. 15, 2006

Nov. 15, 2006

I had made a mistake in my revious post and placed a perdio in a place where a space character was needed: the help text for --apigen, just before the '__package__'. The diff that fixes it is attached. Cheers, P.R.

1 0

Re: [py-dev] Some fixes to minor typos inside strings
by Maciek Fijalkowski Nov. 15, 2006

Nov. 15, 2006

Pierre Rouleau wrote: > Maciek, > > I noticed a couple of typos in the command line strings in > test/cmdline.py and test/defaultconftest.py. > > Here's the diff for the changes I made locally. > > The universal diff output files are attached. > > Very minor changes, but ... > > P.R. > >------------------------------------------------------------------------ > >Index: C:/py-dist/py/test/cmdline.py >=================================================================== >--- C:/py-dist/py/test/cmdline.py (revision 34637) >+++ C:/py-dist/py/test/cmdline.py (working copy) >@@ -32,7 +32,7 @@ > config.option.startserver = True > try: > if config.getinitialvalue('startserver'): >- py.std.warnings.warn("Startserver flag in config is deprecated, use commandline option istead") >+ py.std.warnings.warn("Startserver flag in config is deprecated, use commandline option instead") > except ValueError: > pass > > > >------------------------------------------------------------------------ > >Index: C:/py-dist/py/test/defaultconftest.py >=================================================================== >--- C:/py-dist/py/test/defaultconftest.py (revision 34637) >+++ C:/py-dist/py/test/defaultconftest.py (working copy) >@@ -15,7 +15,7 @@ > py.test.Config.addoptions('general options', > Option('-v', '--verbose', > action="count", dest="verbose", default=0, >- help="increase verbosity"), >+ help="increase verbosity."), > Option('-x', '--exitfirst', > action="store_true", dest="exitfirst", default=False, > help="exit instantly on first error or failed test."), >@@ -24,33 +24,33 @@ > help="disable catching of sys.stdout/stderr output."), > Option('-k', > action="store", dest="keyword", default='', >- help="only run test items matching the given (google-style) keyword expression"), >+ help="only run test items matching the given (google-style) keyword expression."), > Option('-l', '--showlocals', > action="store_true", dest="showlocals", default=False, >- help="show locals in tracebacks (disabled by default)"), >+ help="show locals in tracebacks (disabled by default)."), > Option('', '--pdb', > action="store_true", dest="usepdb", default=False, > help="start pdb (the Python debugger) on errors."), > Option('', '--tb', > action="store", dest="tbstyle", default='long', > type="choice", choices=['long', 'short', 'no'], >- help="traceback verboseness (long/short/no)"), >+ help="traceback verboseness (long/short/no)."), > Option('', '--fulltrace', > action="store_true", dest="fulltrace", default=False, >- help="don't cut any tracebacks (default is to cut)"), >+ help="don't cut any tracebacks (default is to cut)."), > Option('', '--nomagic', > action="store_true", dest="nomagic", default=False, >- help="refrain from using magic as much as possible"), >+ help="refrain from using magic as much as possible."), > Option('', '--collectonly', > action="store_true", dest="collectonly", default=False, >- help="only collect tests, don't execute them. "), >+ help="only collect tests, don't execute them."), > Option('', '--traceconfig', > action="store_true", dest="traceconfig", default=False, >- help="trace considerations of conftest.py files. "), >+ help="trace considerations of conftest.py files."), > Option('', '--apigen', > action="store_true", dest="apigen", default=False, >- help="Generated API docs out of tests. Needs to have defined"\ >- "__package__ for module or overwritten in conftest") >+ help="Generated API docs out of tests. Needs to have defined "\ >+ "__package__ for module or overwritten in conftest.") > ) > py.test.Config.addoptions('test-session related options', > Option('', '--tkinter', >@@ -67,11 +67,11 @@ > help="python executable to run the tests with. "), > Option('-w', '--startserver', > action="store_true", dest="startserver", default=False, >- help="Start HTTP server listening on localhost:8000 for test" >+ help="Start HTTP server listening on localhost:8000 for test." > ), > Option('', '--runbrowser', > action="store_true", dest="runbrowser", default=False, >- help="Run browser to point to your freshly started web server" >+ help="Run browser to point to your freshly started web server." > ), > ) > > > Thx a lot. Commited. It really doesn't matter that the changes are minor, we appreciate any feedback.

1 0

Re: [py-dev] Fwd: Speed of py.test compared to unittest
by Pierre Rouleau Nov. 15, 2006

Nov. 15, 2006

On 11/15/06, Maciek Fijalkowski <fijal(a)genesilico.pl> wrote: > >> > >> I've profiled it a bit. Right now (for me) your tests are about 3x > >> slower that unittests. It still can be improved somehow. > >> > >> Btw: Tests are unfair, cause collection encounters 5k tests for py.test > >> and 13 for unittests, if you rewrite it so they'll be equal it would be > >> better. It's not good excuse anyway ;-) > >> > >> I'll try to work a bit more on it, but I guess achieving 2x in this > >> particular case would be enough. If you still encounter massive > >> slowdowns in different enviroments, do not hesitate to contact us. > >> > > True, the code I sent you is not comparing the exact same tests and it > > is unfair. > > So, as I kept looking into unit test for Python, I found nose and its > > nosetests that essentially does the same as py.test (without having > > the same design goals as the Py lib). I ran the same test on both > > tools. > > > > The results are: > > - py.test tests in [4.30s, 4.17s, 4.28s, 4.42s] > > - nosetests tests in [ 1.48s, 1.43s, 1.40s, 1.50s] > > > > So it looks like achieving a 2x would make it run in the same ballpark > > as nosetests (at least for that code). > > > > I do like py.test error report better than nosetest though. The > > reporting is less verbose. > > > > Btw: it's updated version? (I did some improvements today/yesterday) > No! I re-ran the tests with your latest version and I got faster results! Great! The results are now: [ 1.94s, 1.92s, 1.92s, 1.98s, 1.92s] which is much closer to the nosetest results. Note however, that when I run nosetests, it executes the *exact* same test, which means that the test file (test_roman.py) contains: from py.test import raises and it uses the raises() call to check that exceptions are raised. I am currently looking into nose to see how i could replace the py.test.raises with nose equivalent. Cheers, P.R.

1 0

[py-dev] Fwd: Fwd: Speed of py.test compared to unittest
by Pierre Rouleau Nov. 15, 2006

Nov. 15, 2006

On 11/15/06, Maciek Fijalkowski <fijal(a)genesilico.pl> wrote: > Carl Friedrich Bolz wrote: > > >Pierre Rouleau wrote: > > > > > >>On 11/14/06, Carl Friedrich Bolz <cfbolz(a)gmx.de> wrote: > >> > >> > >>>I am not quite sure I buy that argument since py.test tells you the time > >>>that the test itself took (if you use the -v argument) or that all tests > >>>together take (at the end of each test run). As far as I remember, these > >>>times don't include any startup times. Yes, wall clock time of the whole > >>>test run is quite a bad measure, but you don't have to use it anyway. > >>> > >>>Cheers, > >>> > >>>Carl Friedrich > >>> > >>> > >>> > >>I didn't realize that the time of *each* test is shown when the -v > >>option is used. Thanks for pointing that out. > >> > >> > > > >you're welcome :-) > > > >Note that I didn't mean to imply that it isn't worthwhile to make > >py.test startup time smaller. it's definitely a good thing to do, since > >testing should have low overhead. > > > >Cheers, > > > >Carl Friedrich > >_______________________________________________ > >py-dev mailing list > >py-dev(a)codespeak.net > >http://codespeak.net/mailman/listinfo/py-dev > > > > > I've profiled it a bit. Right now (for me) your tests are about 3x > slower that unittests. It still can be improved somehow. > > Btw: Tests are unfair, cause collection encounters 5k tests for py.test > and 13 for unittests, if you rewrite it so they'll be equal it would be > better. It's not good excuse anyway ;-) > > I'll try to work a bit more on it, but I guess achieving 2x in this > particular case would be enough. If you still encounter massive > slowdowns in different enviroments, do not hesitate to contact us. > True, the code I sent you is not comparing the exact same tests and it is unfair. So, as I kept looking into unit test for Python, I found nose and its nosetests that essentially does the same as py.test (without having the same design goals as the Py lib). I ran the same test on both tools. The results are: - py.test tests in [4.30s, 4.17s, 4.28s, 4.42s] - nosetests tests in [ 1.48s, 1.43s, 1.40s, 1.50s] So it looks like achieving a 2x would make it run in the same ballpark as nosetests (at least for that code). I do like py.test error report better than nosetest though. The reporting is less verbose. py.test:: _______________________________________________________________________________ ________________________ entrypoint: test_bad_roman[0] ________________________ def check(exception_cls, invalid_numeral): print 'Testing : ', invalid_numeral E raises(exception_cls, roman.fromRoman, invalid_numeral) > ExceptionFailure: DID NOT RAISE [D:\dev\python\test\test_roman.py:164] - - - - - - - - - - - - - - [0]: recorded stdout - - - - - - - - - - - - - - - Testing : MMMM _______________________________________________________________________________ ============ tests finished: 5023 passed, 1 failed in 4.17 seconds ============ and the same failure shown by nosetest:: ====================================================================== ERROR: test_roman.test_bad_roman:(<class roman.InvalidRomanNumeralError at 0x00C5DDB0>, 'MMMM') ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\python24\lib\site-packages\nose-0.9.1-py2.4.egg\nose\case.py", line 52, in runTest self.testFunc() File "c:\python24\lib\site-packages\nose-0.9.1-py2.4.egg\nose\loader.py", line 324, in run expr[0](*expr[1:]) File "D:\dev\python\test\test_roman.py", line 164, in check raises(exception_cls, roman.fromRoman, invalid_numeral) File "C:\py-dist\py\test\raises.py", line 39, in raises expr=args, expected=ExpectedException) ExceptionFailure: DID NOT RAISE -------------------- >> begin captured stdout << --------------------- Testing : MMMM --------------------- >> end captured stdout << ---------------------- ---------------------------------------------------------------------- Ran 5024 tests in 1.407s FAILED (errors=1) Cheers, P.R.

1 0

[py-dev] Speed of py.test compared to unittest
by Pierre Rouleau Nov. 15, 2006

Nov. 15, 2006

Hi all! I'v just started looking into py.test. I really like it: it's easier to use and requires less boiler plate code than unittest. Thanks! However, i found that it runs much slower than unittest (at least under some conditions). I was expecting a slowdown because of py.test's nature, but I get results that are 10 to 30 times slower. I ran a test of a simple application that converts roman numeral to integer and vice versa (taken from Mike Pilgrim's http://diveintopython.org/ ). In a test that checks all 4999 possible numbers, unittest performs in about 125 mSeconds, while py.test takes about 3.8 seconds. That's 30 times slower. The unittest and py.test driven scripts perform the same tests. Is this what I should expect? I am running Python 2.4.3 on Win32 machine with py-dist revision 34574 (the latest) from svn. Thanks, Pierre Rouleau

2 1

[py-dev] Fwd: Speed of py.test compared to unittest
by Pierre Rouleau Nov. 15, 2006

Nov. 15, 2006

---------- Forwarded message ---------- From: Pierre Rouleau <prouleau001(a)gmail.com> Date: Nov 14, 2006 4:18 PM Subject: Re: [py-dev] Speed of py.test compared to unittest To: Maciek FijaĹkowski <fijal(a)genesilico.pl> On 11/14/06, Maciek FijaĹkowski <fijal(a)genesilico.pl> wrote: > > Sure. Please do. > Oh, also, as far as startup time is concerned, even though the startup time might be constant, a larger startup time will mask differences in execution time of the code under test. Since a unit test system should optimally be used during development, a startup time close to 0 will show timing impact of the code being tested as it is being modified. And this might be useful during development (even though you might want to measure performance of the module being developed using other tools).

3 4

Re: [py-dev] Speed of py.test compared to unittest
by Maciek FijaĹkowski Nov. 14, 2006

Nov. 14, 2006

Pierre Rouleau wrote: >Hi all! > >I'v just started looking into py.test. I really like it: it's easier >to use and requires less boiler plate code than unittest. Thanks! > >However, i found that it runs much slower than unittest (at least >under some conditions). I was expecting a slowdown because of >py.test's nature, but I get results that are 10 to 30 times slower. > >I ran a test of a simple application that converts roman numeral to >integer and vice versa (taken from Mike Pilgrim's >http://diveintopython.org/ ). >In a test that checks all 4999 possible numbers, unittest performs in >about 125 mSeconds, while py.test takes about 3.8 seconds. That's 30 >times slower. >The unittest and py.test driven scripts perform the same tests. > >Is this what I should expect? > >I am running Python 2.4.3 on Win32 machine with py-dist revision 34574 >(the latest) from svn. > >Thanks, > >Pierre Rouleau >_______________________________________________ >py-dev mailing list >py-dev(a)codespeak.net >http://codespeak.net/mailman/listinfo/py-dev > > I would be happy to run some performance tests on my machine and profiling as well. Please send to me some details (how to access the code ie.). Please also note that a lot of time from that 3.8s is startup time (I guess it's not too much pain to just allow it to be about 2s).

1 0

[py-dev] py lib API
by Maciek FijaĹkowski Nov. 13, 2006

Nov. 13, 2006

I've created some experimental API of pylib (automatically generated). It's very rough by now, so don't expect fancy CSS and JS (at least now), but I'm very interested in any feedback about contents. http://codespeak.net/~fijal/output/ Cheers, Fijal

1 0

[py-dev] User defined exception - py.testing
by Santosh Philip Nov. 12, 2006

Nov. 12, 2006

I am having some trouble with testing user defined exceptions ============== this one works ------------ def anexception(arg): """just raise an exception with ValueError""" if arg == 5: raise ValueError, 'Just a ValueError' def test_anexception(): """py.test for anexception""" import py.test py.test.raises(ValueError, anexception, 5) =============== this one does not work --------------- class MyError(Exception): """pass""" pass def anotherexception(arg): """just raise an exception with ValueError""" if arg == 5: raise MyError, 'its a MyError' def test_anotherexception(): """py.test for anotherexception""" import py.test py.test.raises(ValueError, anotherexception, 5) ================ Are test for user defined exceptions working in py.test or Am I doing something wrong ? when it doesn't work, the output looks like MacBook:~/Documents/doe2python_DSU santosh$ py.test doesntwork.py inserting into sys.path: /Users/santosh/Documents/coolshadowprojects/pydottest/py-dist ============================= test process starts ============================== testing-mode: inprocess executable: /Library/Frameworks/Python.framework/Versions/2.4/Resources/Python.app/ Contents/MacOS/Python (2.4.3-final-0) using py lib: /Users/santosh/Documents/coolshadowprojects/pydottest/py-dist/py <rev 31477> doesntwork.py[1] F ________________________________________________________________________ ________ ______________________ entrypoint: test_anotherexception _______________________ def test_anotherexception(): """py.test for anotherexception""" import py.test > py.test.raises(ValueError, anotherexception, 5) [/Users/santosh/Documents/doe2python_DSU/doesntwork.py:13] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def anotherexception(arg): """just raise an exception with ValueError""" if arg == 5: E raise MyError, 'its a MyError' > MyError: its a MyError [/Users/santosh/Documents/doe2python_DSU/doesntwork.py:8] ________________________________________________________________________ ________ =================== tests finished: 1 failed in 0.02 seconds =================== thanks Santosh

2 1