Mailman 3 October 2010 - pytest-dev

[py-dev] py.test api - Running tests in a module object?
by Sridhar Ratnakumar Nov. 29, 2010

Nov. 29, 2010

I am trying to run py.test on a given module object dynamically. mod = __import__('company.foo.somemod') py.test.run(mod) A quick glance in the py.test source doesn't seem like a trivial thing. IPython introspection did not give me any clue either. Is there a way to do this at all? I did try py.test.cmdline.main(['...']), but that accepts only file path, not the Python module object itself. -srid

2 5

[py-dev] py.test with setuptools
by James Mills Oct. 28, 2010

Oct. 28, 2010

Hey all, Just curious... How do you setup your setup.py so that "python setup.py tests" works ? cheers James -- -- James Mills -- -- "Problems are solved by method"

2 2

Re: [py-dev] py.test to control other testing framework
by holger krekel Oct. 28, 2010

Oct. 28, 2010

Hi Eli, On Thu, Oct 28, 2010 at 14:11 -0700, Ateljevich, Eli wrote: > Thanks Holger. I said wanted to count assertions but I was just being careless -- counting test failures and passes the usual way is fine. > > You seem to understand my timing issue by asking "do I know how many tests there are"? I'll ask the cxxtest folks, but I haven't seen it in their usage examples and there is no obvious --count-tests type of flag. Python would see each cxxtest suite as a monolith. > > One alternative is to lie to the py.test reporting mechanism. If I understand that code snippet you pointed out, I get a shot at this in repr_failure(self, excinfo). Is excinfo an opportunity to manipulate test attributes enough to increase the total, success and fail counts: > 1. Collector discovers cxxtests > 2. Runner runs them > 3. Runner parses the success/fail results and stashes them > 4. Runner lies in repr_failure(self,excinfo), upping the counts > Even if this is possible, it seems like at most I can do this on failure -- the number of tests and successes would be hard to count correctly. You can't really "lie in repr_failure" regarding test counts. We could maybe think about allowing multiple reports per test item. Haven't tried this and am a bit skeptical about it producing internal problems. It might be surprising for other plugins like the junitxml plugin or the terminal plugin itself which assume they get one report per collected test item IIRC. However, it's worth an experiment i guess especially since Ronny (another py.test hacker and heavy user) recently arrived at a similar need, it seems. > The other alternative is to run and count the tests at collect time: > 1. Collector runs the cxxtests > 2. Collector parses the test results and emits shadow tests that trivially fail, succeed or have an error. > 3. Test runner runs the shadow tests. > I am wary of running cxxtests during "collect", but I can't really say why. > > What do you think? Right, it doesn't look completely fitting. Running tests at collect time sounds wrong. If i were you I'd probably first try to collect one test item per test file and parse the log/result file, raise a failure if anything failed and show details if any test/assertion fails. I'd probably first perfect this (including properly and robustly parsing the log) and see how the resulting testing process feels in practise. The involved coding effort is worthwhile in any case. If allowing multiple results still feels important maybe I can try / experiment with the above idea (somehow allowing multiple results per one test item) or you could try this: a test run could create a cache file which will be found by subsequent collections which will then create multiple items and not just one. The items can - at runtest() time - easily share the same result log from a cxx-run. cheers, holger P.S.: let's keep the discussion on py-dev as this may be interesting for future reference or other people. > Thanks so much for following up with this!!! > > Eli > > > > ________________________________________ > From: holger krekel [holger(a)merlinux.eu] > Sent: Wednesday, October 27, 2010 11:52 PM > To: Ateljevich, Eli > Cc: py-dev(a)codespeak.net > Subject: Re: [py-dev] py.test to control other testing framework > > Hi Eli, > > On Wed, Oct 27, 2010 at 17:00 -0700, Ateljevich, Eli wrote: > > On another list, I asked Holger about how to use py.test as a wrapper to control non-python tests. He referred me to this example: > > self-contained yaml example<http://codespeak.net/~hpk/pytest/example/nonpython.html> > > > > This got me going on the issue of collecting and running the tests. > > > > I have a follow-up question about aggregating results from these "foreign tests". I am using cxxunit, but the specifics are not important to my question. Each non-python test involves multiple asserts, failures and possibly errors. These test results could be reported in any of the usual formats (logs, stdout, JUnit xml format, custom). > > (sidenote: I guess you are aware of the --junitxml option.) > > > My question is this: is there a good way to track assert pass/fail counts correctly in py.test? > > > > One crude idea is to have cxxtest print out its assert attempts, passes and failures to a log, parse the log and then deliberately pass and fail a correct number of "shadow assertions" in python using the same log messages. Is there a more direct way? > > Parsing a log sounds right. However, there currently is no notion of "counting > asserts" in py.test (or other popular python testing frameworks i am aware of). > > Do you happen to have a way to find out the number of test functions/cases > ahead of running a test file? And to instruct cxxtest to run a particular > function? > > If so you could map cxx test functions to py.test Items and get > more fine-granular "." printing and error representation. > > Otherwise i guess you can only represent the whole cxxtest file as > one test item. Also because of other scenarios i am wondering how/if > to help this situation, btw. > > cheers, > holger --

1 0

[py-dev] py.test to control other testing framework
by Ateljevich, Eli Oct. 28, 2010

Oct. 28, 2010

On another list, I asked Holger about how to use py.test as a wrapper to control non-python tests. He referred me to this example: self-contained yaml example<http://codespeak.net/~hpk/pytest/example/nonpython.html> This got me going on the issue of collecting and running the tests. I have a follow-up question about aggregating results from these "foreign tests". I am using cxxunit, but the specifics are not important to my question. Each non-python test involves multiple asserts, failures and possibly errors. These test results could be reported in any of the usual formats (logs, stdout, JUnit xml format, custom). My question is this: is there a good way to track assert pass/fail counts correctly in py.test? One crude idea is to have cxxtest print out its assert attempts, passes and failures to a log, parse the log and then deliberately pass and fail a correct number of "shadow assertions" in python using the same log messages. Is there a more direct way? Thanks, Eli

2 1

[py-dev] New issue 130 in py-trunk: py.process.cmdexec fails if the out/err contains non-ascii characters
by issues-noreply＠bitbucket.org Oct. 20, 2010

Oct. 20, 2010

--- you can reply above this line --- New issue 130: py.process.cmdexec fails if the out/err contains non-ascii characters http://bitbucket.org/hpk42/py-trunk/issue/130/pyprocesscmdexec-fails-if-the… Antonio Cuni / antocuni on Wed, 20 Oct 2010 11:30:44 +0200: Description: Consider the following file trouble.py, which just output a non-ascii character to stdout: {{{ # -*- encoding: utf-8 -*- trouble = u'à' print trouble.encode('utf-8') }}} py.process.cmdexec fails if we try to run it: {{{ >>> import py >>> py.process.cmdexec('python trouble.py') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/antocuni/pypy/misc/py-trunk/py/_process/cmdexec.py", line 32, in cmdexec raise ExecutionFailed(status, status, cmd, out, err) py.process.cmdexec.Error: ExecutionFailed: 1 python trouble.py Traceback (most recent call last): File "trouble.py", line 3, in <module> print trouble UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 0: ordinal not in range(128) }}} This happens because cmdexec try to decode stderr and stdout using sys.stdout.encoding or sys.getdefaultencoding(), which not necessarly match the output encoding of the program. Real life use-case: a pypy test invokes gcc, which uses utf-8 characters for quotes. The test fails because of the exception inside cmdexec, even if the test itself completely ignores the stderr. Responsible: hpk42 -- This is an issue notification from bitbucket.org. You are receiving this either because you are the owner of the issue, or you are following the issue.

1 0

[py-dev] Detect whether py.test is running
by Antonio Cuni Oct. 20, 2010

Oct. 20, 2010

Hi, I need a way to distinguish whether the current module is being imported because we are running the tests, of whether we are running the main program. I know that the correct way would be to inject the information from the top, but I'm using a framework which relies on some global state to store the various options, and I have to deal with it. Anyway, is there any official/recommended way to do it? What I'm doing right now is this (py 1.3.1): if py.test.config.__dict__ == {}: # py.test is NOT running ... else: # py.test is running ciao, Anto

3 6

[py-dev] pytest / pylib / tox release splitting plan
by holger krekel Oct. 15, 2010

Oct. 15, 2010

Hi all, here are some py.test/pylib/tox split release related stories i have been thinking about. Idea is to allow all kinds of installation / upgrade situations to succeed. My goal is to have "tox" only pull in the new small and mean "pylib" and not the whole pytest package. Another is to have a small mostly self-contained "pytest" package. And a third to seperate out the helper tools into a "pycmd" package. Anyone who is interested in release management might want to read this, and maybe help me see the light or be inspired ... all others can just hope i get this right and don't mess up :) consider this situation: pip install py # you currently get sumo py==1.3.4, for using py.test pip install pylib # will be pylib==2.0.0 this would lead to "import py" coming from py==1.3.4 because the distribution 'py' "owns py" and came first. You'd also see funny tracebacks when invoking py.test. BAD. My planned fix is to provide a py==1.4.0 which has _no_ packages and have pylib depend on the exact "py==1.4.0" version. Then pip install pylib will lead to "import py" coming from pylib-2.0.0 because the py==1.4.0 is an upgrade to 1.3.4, thus removes its "py", package, no interference. Good. Now consider a tox user, typing:: pip install tox # let's assume this is 0.8 tox has a requirement of "py" and if this drags py==1.4.0 - BUMM! the tox installs breaks, because when it "import py" nothing is there. My planned fix is to provide *another* release "py==1.4.1" which will depend on pylib-2.0 and pytest-2.0 (and pycmd-1.0) so that people blindly installing tox like above will be happy and people doing pip install -U py and expecting to get py.test (because they are reading an old README or blog post) will also be happy and use the newest version of pytest/pylib. Finally, somebody starting a fresh installation with pip install pytest will just drag in pylib-2.0, which drags in py==1.4.0 which requires pylib-2.0 ... SIMPLE, good. You might wonder why i care so much and don't simply tell people "make sure you uninstall py first" or something like this. Reason: many people are using the above packages in Continous Integration situations and going to all kinds of masters, slaves and Job configurations to ensure removal of packages and such is not fun. And not everybody reads release messages anyway ... or understands what is installed in their environments - judging from myself :) best, holger

1 0

[py-dev] installing new py.test / new pypi.testrun.org
by holger krekel Oct. 14, 2010

Oct. 14, 2010

Hi, i created a simple PyPI server instance as a playground and to make it easier to test interdependent packagages i am working on. See http://pypi.testrun.org You can use it to install the latest in-development version of pytest, tox, pylib, pycmd, pytest-xdist etc. (*) with e. g.: pip install -i http://pypi.testrun.org pytest or tox:: pip install -i http://pypi.testrun.org tox These are _development_ versions so they might break more easily. But they also fix a number of bugs and provide some new features :) cheers, holger

1 0

[py-dev] do you mind the issue-postings here?
by holger krekel Oct. 12, 2010

Oct. 12, 2010

Hi all, am wondering: do you mind seeing new bitbucket issue creations here on the mailing list? If so i'll create a new mailing list pypy-issue or so and have things rather sent there. Just let me know. Personally i find it a bit unfortuante that the many closed/resolved issues are not notified to this list as well :) cheers, holger

2 1

[py-dev] New issue 129 in py-trunk: Test file names of the form foo_bar___init__.py error in 1.3.2, 1.3.3 and 1.3.4
by issues-noreply＠bitbucket.org Oct. 12, 2010

Oct. 12, 2010

--- you can reply above this line --- New issue 129: Test file names of the form foo_bar___init__.py error in 1.3.2, 1.3.3 and 1.3.4 http://bitbucket.org/hpk42/py-trunk/issue/129/test-file-names-of-the-form-f… trevorbaca on Tue, 12 Oct 2010 17:32:46 +0200: Description: Hi, Starting in 1.3.2 test file names *ending with* the string '__init__.py' cause py.test to fail. This would seem to relate to #119 (fixed in 1.3.4). But this version of the problem persists in the 1.3.4 public release. In the example below, the file foo_bar___init__.py contains just a single print statement: Desktop$ py.test foo_bar__init__.py ======================================= test session starts ======================================== platform darwin -- Python 2.6.1 -- pytest-1.3.4 test path 1: foo_bar__init__.py foo_bar__init__.py E ============================================== ERRORS ============================================== __________________ ERROR collecting /Users/trevorbaca/Desktop/foo_bar__init__.py ___________________ /Library/Python/2.6/site-packages/py-1.3.4-py2.6.egg/py/_plugin/pytest_runner.py:34: in pytest_make_collect_report > result = collector._memocollect() /Library/Python/2.6/site-packages/py-1.3.4-py2.6.egg/py/_test/collect.py:230: in _memocollect > return self._memoizedcall('_collected', self.collect) /Library/Python/2.6/site-packages/py-1.3.4-py2.6.egg/py/_test/collect.py:104: in _memoizedcall > res = function() /Library/Python/2.6/site-packages/py-1.3.4-py2.6.egg/py/_test/pycollect.py:82: in collect > dicts = [getattr(self.obj, '__dict__', {})] /Library/Python/2.6/site-packages/py-1.3.4-py2.6.egg/py/_test/pycollect.py:17: in fget > self._obj = obj = self._getobj() /Library/Python/2.6/site-packages/py-1.3.4-py2.6.egg/py/_test/pycollect.py:140: in _getobj > return self._memoizedcall('_obj', self._importtestmodule) /Library/Python/2.6/site-packages/py-1.3.4-py2.6.egg/py/_test/collect.py:104: in _memoizedcall > res = function() /Library/Python/2.6/site-packages/py-1.3.4-py2.6.egg/py/_test/pycollect.py:145: in _importtestmodule > mod = self.fspath.pyimport(ensuresyspath=True) /Library/Python/2.6/site-packages/py-1.3.4-py2.6.egg/py/_path/local.py:539: in pyimport > if not self.samefile(modfile): /Library/Python/2.6/site-packages/py-1.3.4-py2.6.egg/py/_path/local.py:73: in samefile > return py.error.checked_call(os.path.samefile, str(self), str(other)) /Library/Python/2.6/site-packages/py-1.3.4-py2.6.egg/py/_error.py:80: in checked_call > raise cls("%s%r" % (func.__name__, args)) E ENOENT: [No such file or directory]: samefile('/Users/trevorbaca/Desktop/foo_bar__init__.py', '/Users/trevorbaca/Desktop/foo_ba') ===================================== 1 error in 0.09 seconds ====================================== This matters for us at www.projectabjad.org because we author tests for the initializers of all of our classes in files named as FooBarClass___init__.py. Trevor. Responsible: hpk42 -- This is an issue notification from bitbucket.org. You are receiving this either because you are the owner of the issue, or you are following the issue.

1 0