[Numpy-discussion] testing numpy with downstream testsuites (was: Re: Notes from the numpy dev meeting at scipy 2015)

[Matthew Brett]

Hi,

On Wed, Aug 26, 2015 at 7:59 AM, Nathaniel Smith <njs at pobox.com> wrote:
> [Popping this off to its own thread to try and keep things easier to follow]
>
> On Tue, Aug 25, 2015 at 9:52 AM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
>>>   - Lament: it would be really nice if we could get more people to
>>>     test our beta releases, because in practice right now 1.x.0 ends
>>>     up being where we actually the discover all the bugs, and 1.x.1 is
>>>     where it actually becomes usable. Which sucks, and makes it
>>>     difficult to have a solid policy about what counts as a
>>>     regression, etc. Is there anything we can do about this?
>>
>> Just a note in here - have you all thought about running the test suites for
>> downstream projects as part of the numpy test suite?
>
> I don't think it came up, but it's not a bad idea! The main problems I
> can foresee are:
> 1) Since we don't know the downstream code, it can be hard to
> interpret test suite failures. OTOH for changes we're uncertain of we
> already do often end up running some downstream test suites by hand,
> so it can only be an improvement on that...
> 2) Sometimes everyone including downstream agrees that breaking
> something is actually a good idea and they should just deal, but what
> do you do then?
>
> These both seem solvable though.
>
> I guess a good strategy would be to compile a travis-compatible wheel
> of $PACKAGE version $latest-stable against numpy 1.x, and then in the
> 1.(x+1) development period numpy would have an additional travis run
> which, instead of running the numpy test suite, instead does:
>   pip install .
>   pip install $PACKAGE-$latest-stable.whl
>   python -c 'import package; package.test()' # adjust as necessary
> ? Where $PACKAGE is something like scipy / pandas / astropy / ...
> matplotlib would be nice but maybe impractical...?
>
> Maybe someone else will have objections but it seems like a reasonable
> idea to me. Want to put together a PR? Asides from fame and fortune
> and our earnest appreciation, your reward is you get to make sure that
> the packages you care about are included so that we break them less
> often in the future ;-).

One simple way to get going would be for the release manager to
trigger a build from this repo:

https://github.com/matthew-brett/travis-wheel-builder

This build would then upload a wheel to:

http://travis-wheels.scikit-image.org/

The upstream packages would have a test grid which included an entry
with something like:

pip install -f http://travis-wheels.scikit-image.org --pre numpy

Cheers,

Matthew