On Mon, May 13, 2019 at 3:02 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:


On Sun, May 12, 2019 at 4:41 PM Charles R Harris <charlesr.harris@gmail.com> wrote:


On Sun, May 12, 2019 at 7:27 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:


On Sun, May 12, 2019 at 3:56 PM Charles R Harris <charlesr.harris@gmail.com> wrote:


On Sun, May 12, 2019 at 6:33 AM Julian Taylor <jtaylor.debian@googlemail.com> wrote:
On 12.05.19 14:58, Charles R Harris wrote:
> Hi All,
>
> NumPy currently distinguishes between release and development versions
> when running tests. Is there a good reason to continue this practice? I
> ask, because with the last pytest release it would be convenient to
> always include `pytest.ini ` so that we can register markers. The
> presence of `pytest.ini` is how we distinguish betweendevelopment from
> release for testing purposes.
>

One difference between development and release builds was that in
development releases numpy.testing throws errors on floating point
exceptions while the release version it did not.

I'd prefer to keep this behavior. It's not clear to me if the proposal is to change the behavior or not.

If that is still the case removing the distinction could require a lot
of changes in upstream test suites that are not regularly run against
development builds.

The motivation is not quite clear to me, can you please elaborate on
what you want to do.

NumPy pytest testing is NumPy specific and not used downstream like our nose testing framework was, so I don't see why that should affect other projects. What motivates this question is that the new version of pytest released yesterday raises warnings for non-registered markers, `pytest.mark.slow` in particular, and that was causing CI failures. The easiest way to register a mark is using `pytest.ini`, but we currently don't include that in released wheels, only in source releases.

Adding a pytest.ini file in wheels should be perfectly fine I think,


It is the absence of pytest.ini that makes it a release, for that is the file that turns warnings into errors.

Why don't we just always keep pytest.ini, and move the settings for warnings-to-errors to runtests.py or tools/travis_test.sh?

It's not important how we check, all we need is some mechanism to prevent new warnings creeping in. Same as we set -Wall for building in CI.


I just checked that current wheels show the warnings when `numpy.test()` is run with latest pytest.  However, moving the `pytest.ini` file into the `numpy` directory is tricky, as we need to tell pytest where to find the installed file (-c option). The simplest short time solution is to ignore the warning, but long term I'm worried that the warning will become an error as pytest is doing this because they want to clean up the their implementation. Ideally there would be a better way to register the marks. Note that we currently deal with the missing `pytest.ini` in wheels by duplicating the warnings filters in `_pytesttester.py` using the pytest command line.

Adding the `error` filter in the command line worries me, as I'm not sure how the priorities work out, the command line is now appended to the contents of `pytest.ini`.

Chuck