[IPython-dev] Dropping our parametric tests system

[Fernando Perez]

Hi all,

On Fri, Aug 30, 2013 at 3:32 PM, Paul Ivanov <pi at berkeley.edu> wrote:

> I agree that we shouldn't need an IPEP - I don't think *anyone*
> is counting on our test suite to be a particular way - it is not
> a part of the codebase that we document (outside of API docs) or
> advertise to outside consumers.
>
> I'm glad we'll finallly stop using such tricky (and fallible)
> machinary, though I admit I'll miss the extra dots... Though with
> the potential change to our testing infrastructure in the future,
> as discussed at this week's Lab Meeting On Air [1], we might not
> have the dots in the future anyway.

Just for the record, we just had a chat here also with Min, who was
perhaps the last happy user of the system.

I also don't think an IPEP is necessary, and the main reason to get
rid of this is ultimately its brittleness.  As much as I thought it
was a good idea at the time to do this (because it does have its
benefits), over time I've had it break on me at the worst possible
time: when you are debugging, the last thing you want to be fighting
is your test suite. Paraphrasing JWZ:

you have a test problem and you think "aha! I'll write some parametric
tests". Now you have two problems.

When your test suite breaks while you're trying to hunt down a bug,
you get stopped dead on your tracks and are forced to switch gears
from the problem you were solving first to debugging the testing
machinery itself. And in this particular case, that means debugging
the horrifically convoluted intersection of our own parametric
extensions, nose and unittest.  I have done it and trust me, you don't
want to.

Min mentioned that part of his attachment to the system was that it
seemed like such a natural and simple thing to do that it *should* be
easy and robust. I totally agree with that sentiment, but sadly the
internal architecture of unittest is so absurdly nasty (right up there
with the beauty of distutils) that it makes what should be a very
simple job a very difficult one. As an indicator of the fact that this
is *not* trivial, when I first wrote that code I also filed an issue
on Python:

http://bugs.python.org/issue7897

and it has spawned two more:

http://bugs.python.org/issue12600
http://bugs.python.org/issue16997

spanning over 3 1/2 years of work  and discussion.  And the current
resolution they are offering for 3.4 doesn't actually satisfy all the
problems raised...

In any case, while I don't think we need an IPEP for this, as Min
pointed out having the discussion in public is good to record the
decision and our rationale for it.

Cheers,

f