[IPython-dev] correct test-suite

[Fernando Perez]

Hi Tom,

On Sun, Jul 18, 2010 at 8:14 AM, Thomas Spura <tomspur at fedoraproject.org> wrote:
> There is now a Makefile, so it's nicer to run repetive tasks in the
> repository, but currently there is only 'make test-suite', which should
> run the test suite.
> (now = in branch my_fix_test_suite at github:
> http://github.com/tomspur/ipython/commits/my_fix_test_suite)

Unfortunately this approach does not work if IPython isn't installed
system-wide, because it breaks the twisted tests:

===============================================================================
[ERROR]: IPython.kernel

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/twisted/trial/runner.py",
line 651, in loadByNames
    things.append(self.findByName(name))
  File "/usr/lib/python2.6/dist-packages/twisted/trial/runner.py",
line 461, in findByName
    return reflect.namedAny(name)
  File "/usr/lib/python2.6/dist-packages/twisted/python/reflect.py",
line 471, in namedAny
    raise ObjectNotFound('%r does not name an object' % (name,))
twisted.python.reflect.ObjectNotFound: 'IPython.kernel' does not name an object
-------------------------------------------------------------------------------

as well as some others.  And the manual sys.path manipulations done here:

http://github.com/tomspur/ipython/commit/7abd52e0933aa57a082db4623c791630bc0671ea#L0R51

should be avoided in production code, which iptest.py is.

I'm not opposed to having a top-level makefile, but the one in your
commit can't be merged for this reason.  Additionally, it makes
targets for things like tar/zip generation that should be done as
distutils commands instead (and they are done by some of the scripts
in tools/).

In practice, I find that simply having the current working IPython/
directory symlinked in my PYTHONPATH lets me run the iptest script at
any time without further PYTHONPATH manipulations.

I'm happy to find a solution to running the test suite without
installing, but this one doesn't seem to work robustly (and I'd
already backed off a while ago some more hackish things I'd tried,
precisely for being too brittle).

Until we have a really clean solution, we'll have a test suite that
can only be run from an installed IPython (or equivalently, a setup
that finds the source tree transparently, which is what I use).

> One failing test pointed out, that there is a programming error in
> IPython/Shell.py and is now corrected in this commit:
> http://github.com/tomspur/ipython/commit/7e7988ee9e7c35b2e5302725ebdf6c22135f334e

This one I did just cherry-pick and push, as it was really a clean
bug, thanks for the fix:

http://github.com/ipython/ipython/commit/a469f3d77cf794b33ac20cf9d3f2246387423808

> But now, there is a problem with test: "Test that object's __del__
> methods are called on exit." in IPython/core/tests/test_run.py:146.

I think that's all caused by the problems you are seeing from your
method of running the tests.  On my system, all tests do pass cleanly
right now:

**********************************************************************
Test suite completed for system with the following information:
IPython version: 0.11.alpha1.git
BZR revision   : 0
Platform info  : os.name -> posix, sys.platform -> linux2
               : Linux-2.6.32-24-generic-i686-with-Ubuntu-10.04-lucid
Python info    : 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)
[GCC 4.4.3]

Tools and libraries available at test time:
   curses foolscap gobject gtk pexpect twisted wx wx.aui zope.interface

Tools and libraries NOT available at test time:
   objc

Ran 10 test groups in 46.446s

Status:
OK
####

I also commented on your bundled_libs branch: it can't be merged
because it also breaks most of the Twisted tests.  Until the test
suite passes 100% we can't merge those changes, though I do very much
like the idea of better organizing externals, so I hope you can sort
out the issue.  Do let us know as soon as you can fix those and we can
try again.

So I think right now we have merged everything that is mergeable from
you, right?  Please go ahead and file pull requests again if you do
update these (since those trigger an email and that makes it easier to
keep track of what's been done).

Thanks a lot for your interest and help!

Cheers,

f