[Numpy-discussion] Many failing doctests - release blocker? Enable for default test runs?

[Ralf Gommers]

On Mon, Dec 24, 2012 at 9:15 AM, Matthew Brett <matthew.brett at gmail.com>wrote:

> Hi,
>
> On Sun, Dec 23, 2012 at 8:53 PM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> >
> >
> >
> > On Sun, Dec 23, 2012 at 7:54 PM, Ondřej Čertík <ondrej.certik at gmail.com>
> > wrote:
> >>
> >> Hi Matthew,
> >>
> >> On Sat, Dec 22, 2012 at 9:40 PM, Matthew Brett <matthew.brett at gmail.com
> >
> >> wrote:
> >> > Hi,
> >> >
> >> > I noticed that enabling the doctests on the 1.7.x maintenance branch
> >> > caused lots and lots of doctest failures.
> >> >
> >> > (np-devel)[mb312 at blair ~/dev_trees]$ python -c 'import numpy as np;
> >> > np.test(doctests=True)'
> >> > 1.7.0rc1.dev-1e8fcdf
> >> > Running unit tests and doctests for numpy
> >> > NumPy version 1.7.0rc1.dev-1e8fcdf
> >> > NumPy is installed in
> >> > /Users/mb312/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy
> >> > Python version 2.6.6 (r266:84374, Aug 31 2010, 11:00:51) [GCC 4.0.1
> >> > (Apple Inc. build 5493)]
> >> > nose version 1.1.2
> >> > ...
> >> > Ran 3839 tests in 59.928s
> >> >
> >> > FAILED (KNOWNFAIL=4, SKIP=4, errors=23, failures=175)
> >> >
> >> > The doctests also throw up somewhere round 10 matplotlib plots, so
> >> > presumably those would fail as well on a machine without a display
> >> > without forcing the import of an 'Agg' backend or similar.
> >> >
> >> > I have never checked the doctests on Python 3.  Has anyone run those
> >> > recently?
> >> >
> >> > For the projects I work on most, we enable doctests for the default
> >> > test run - as in 'doctests=True' by default in the numpy testing
> >> > machinery.   Do ya'll see any disadvantage in doing that for numpy?
> >
> >
> > Yes, I do. The doctest framework and reproducibility of reprs across
> Python
> > versions and platforms are too poor to do this. And failing tests give
> new
> > users a bad impression of the quality of numpy.
>
> I believe the repr problems are fairly easily soluble by using minor
> extensions to the current numpy doctest machinery.  I think I was the
> last person to do big modifications to that bit of the numpy codebase
> and I've been using small tweaks to that framework to run cross
> version and cross platform doctest runs by default for a while on lots
> of numpy stuff in nipy:
>
> https://github.com/nipy/nipy/blob/master/nipy/testing/doctester.py#L155
>

My experience is different, but I'm happy to be proven wrong. Let's first
see it running on all Python versions on Travis without issues for a while,
then consider turning it on by default.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20121224/6b6542b5/attachment.html>