[Numpy-discussion] Many failing doctests - release blocker? Enable for default test runs?
matthew.brett at gmail.com
Mon Dec 24 03:15:42 EST 2012
On Sun, Dec 23, 2012 at 8:53 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
> On Sun, Dec 23, 2012 at 7:54 PM, Ondřej Čertík <ondrej.certik at gmail.com>
>> Hi Matthew,
>> On Sat, Dec 22, 2012 at 9:40 PM, Matthew Brett <matthew.brett at gmail.com>
>> > Hi,
>> > I noticed that enabling the doctests on the 1.7.x maintenance branch
>> > caused lots and lots of doctest failures.
>> > (np-devel)[mb312 at blair ~/dev_trees]$ python -c 'import numpy as np;
>> > np.test(doctests=True)'
>> > 1.7.0rc1.dev-1e8fcdf
>> > Running unit tests and doctests for numpy
>> > NumPy version 1.7.0rc1.dev-1e8fcdf
>> > NumPy is installed in
>> > /Users/mb312/.virtualenvs/np-devel/lib/python2.6/site-packages/numpy
>> > Python version 2.6.6 (r266:84374, Aug 31 2010, 11:00:51) [GCC 4.0.1
>> > (Apple Inc. build 5493)]
>> > nose version 1.1.2
>> > ...
>> > Ran 3839 tests in 59.928s
>> > FAILED (KNOWNFAIL=4, SKIP=4, errors=23, failures=175)
>> > The doctests also throw up somewhere round 10 matplotlib plots, so
>> > presumably those would fail as well on a machine without a display
>> > without forcing the import of an 'Agg' backend or similar.
>> > I have never checked the doctests on Python 3. Has anyone run those
>> > recently?
>> > For the projects I work on most, we enable doctests for the default
>> > test run - as in 'doctests=True' by default in the numpy testing
>> > machinery. Do ya'll see any disadvantage in doing that for numpy?
> Yes, I do. The doctest framework and reproducibility of reprs across Python
> versions and platforms are too poor to do this. And failing tests give new
> users a bad impression of the quality of numpy.
I believe the repr problems are fairly easily soluble by using minor
extensions to the current numpy doctest machinery. I think I was the
last person to do big modifications to that bit of the numpy codebase
and I've been using small tweaks to that framework to run cross
version and cross platform doctest runs by default for a while on lots
of numpy stuff in nipy:
More information about the NumPy-Discussion