[Popping this off to its own thread to try and keep things easier to follow]
On Tue, Aug 25, 2015 at 9:52 AM, Nathan Goldbaum
- Lament: it would be really nice if we could get more people to test our beta releases, because in practice right now 1.x.0 ends up being where we actually the discover all the bugs, and 1.x.1 is where it actually becomes usable. Which sucks, and makes it difficult to have a solid policy about what counts as a regression, etc. Is there anything we can do about this?
Just a note in here - have you all thought about running the test suites for downstream projects as part of the numpy test suite?
I don't think it came up, but it's not a bad idea! The main problems I can foresee are: 1) Since we don't know the downstream code, it can be hard to interpret test suite failures. OTOH for changes we're uncertain of we already do often end up running some downstream test suites by hand, so it can only be an improvement on that... 2) Sometimes everyone including downstream agrees that breaking something is actually a good idea and they should just deal, but what do you do then? These both seem solvable though. I guess a good strategy would be to compile a travis-compatible wheel of $PACKAGE version $latest-stable against numpy 1.x, and then in the 1.(x+1) development period numpy would have an additional travis run which, instead of running the numpy test suite, instead does: pip install . pip install $PACKAGE-$latest-stable.whl python -c 'import package; package.test()' # adjust as necessary ? Where $PACKAGE is something like scipy / pandas / astropy / ... matplotlib would be nice but maybe impractical...? Maybe someone else will have objections but it seems like a reasonable idea to me. Want to put together a PR? Asides from fame and fortune and our earnest appreciation, your reward is you get to make sure that the packages you care about are included so that we break them less often in the future ;-). -n -- Nathaniel J. Smith -- http://vorpus.org