[Numpy-discussion] Release vs development testing.

[Charles R Harris]

On Mon, May 13, 2019 at 11:25 AM Charles R Harris <charlesr.harris at gmail.com>
wrote:

>
>
> On Mon, May 13, 2019 at 3:02 AM Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
>
>>
>>
>> On Sun, May 12, 2019 at 4:41 PM Charles R Harris <
>> charlesr.harris at gmail.com> wrote:
>>
>>>
>>>
>>> On Sun, May 12, 2019 at 7:27 AM Ralf Gommers <ralf.gommers at gmail.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Sun, May 12, 2019 at 3:56 PM Charles R Harris <
>>>> charlesr.harris at gmail.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Sun, May 12, 2019 at 6:33 AM Julian Taylor <
>>>>> jtaylor.debian at googlemail.com> wrote:
>>>>>
>>>>>> On 12.05.19 14:58, Charles R Harris wrote:
>>>>>> > Hi All,
>>>>>> >
>>>>>> > NumPy currently distinguishes between release and development
>>>>>> versions
>>>>>> > when running tests. Is there a good reason to continue this
>>>>>> practice? I
>>>>>> > ask, because with the last pytest release it would be convenient to
>>>>>> > always include `pytest.ini ` so that we can register markers. The
>>>>>> > presence of `pytest.ini` is how we distinguish betweendevelopment
>>>>>> from
>>>>>> > release for testing purposes.
>>>>>> >
>>>>>>
>>>>>> One difference between development and release builds was that in
>>>>>> development releases numpy.testing throws errors on floating point
>>>>>> exceptions while the release version it did not.
>>>>>>
>>>>>
>>>> I'd prefer to keep this behavior. It's not clear to me if the proposal
>>>> is to change the behavior or not.
>>>>
>>>> If that is still the case removing the distinction could require a lot
>>>>>> of changes in upstream test suites that are not regularly run against
>>>>>> development builds.
>>>>>>
>>>>>> The motivation is not quite clear to me, can you please elaborate on
>>>>>> what you want to do.
>>>>>>
>>>>>
>>>>> NumPy pytest testing is NumPy specific and not used downstream like
>>>>> our nose testing framework was, so I don't see why that should affect other
>>>>> projects. What motivates this question is that the new version of pytest
>>>>> released yesterday raises warnings for non-registered markers,
>>>>> `pytest.mark.slow` in particular, and that was causing CI failures. The
>>>>> easiest way to register a mark is using `pytest.ini`, but we currently
>>>>> don't include that in released wheels, only in source releases.
>>>>>
>>>>
>>>> Adding a pytest.ini file in wheels should be perfectly fine I think,
>>>>
>>>>
>>> It is the absence of pytest.ini that makes it a release, for that is the
>>> file that turns warnings into errors.
>>>
>>
>> Why don't we just always keep pytest.ini, and move the settings for
>> warnings-to-errors to runtests.py or tools/travis_test.sh?
>>
>> It's not important how we check, all we need is some mechanism to prevent
>> new warnings creeping in. Same as we set -Wall for building in CI.
>>
>>
> I just checked that current wheels show the warnings when `numpy.test()`
> is run with latest pytest.  However, moving the `pytest.ini` file into the
> `numpy` directory is tricky, as we need to tell pytest where to find the
> installed file (-c option). The simplest short time solution is to ignore
> the warning, but long term I'm worried that the warning will become an
> error as pytest is doing this because they want to clean up the their
> implementation. Ideally there would be a better way to register the marks.
> Note that we currently deal with the missing `pytest.ini` in wheels by
> duplicating the warnings filters in `_pytesttester.py` using the pytest
> command line.
>
> Adding the `error` filter in the command line worries me, as I'm not sure
> how the priorities work out, the command line is now appended to the
> contents of `pytest.ini`.
>
>
Note that this is also a problem when numpy is installed with `setup.py
install`. Things work now for runtests because `pytest.ini` is in the
directory from which the tests are run.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190513/b6fa5203/attachment-0001.html>