Release vs development testing.
Hi All, NumPy currently distinguishes between release and development versions when running tests. Is there a good reason to continue this practice? I ask, because with the last pytest release it would be convenient to always include `pytest.ini ` so that we can register markers. The presence of `pytest.ini` is how we distinguish betweendevelopment from release for testing purposes. Chuck
On 12.05.19 14:58, Charles R Harris wrote:
Hi All,
NumPy currently distinguishes between release and development versions when running tests. Is there a good reason to continue this practice? I ask, because with the last pytest release it would be convenient to always include `pytest.ini ` so that we can register markers. The presence of `pytest.ini` is how we distinguish betweendevelopment from release for testing purposes.
One difference between development and release builds was that in development releases numpy.testing throws errors on floating point exceptions while the release version it did not. If that is still the case removing the distinction could require a lot of changes in upstream test suites that are not regularly run against development builds. The motivation is not quite clear to me, can you please elaborate on what you want to do.
On Sun, May 12, 2019 at 6:33 AM Julian Taylor <jtaylor.debian@googlemail.com> wrote:
On 12.05.19 14:58, Charles R Harris wrote:
Hi All,
NumPy currently distinguishes between release and development versions when running tests. Is there a good reason to continue this practice? I ask, because with the last pytest release it would be convenient to always include `pytest.ini ` so that we can register markers. The presence of `pytest.ini` is how we distinguish betweendevelopment from release for testing purposes.
One difference between development and release builds was that in development releases numpy.testing throws errors on floating point exceptions while the release version it did not. If that is still the case removing the distinction could require a lot of changes in upstream test suites that are not regularly run against development builds.
The motivation is not quite clear to me, can you please elaborate on what you want to do.
NumPy pytest testing is NumPy specific and not used downstream like our nose testing framework was, so I don't see why that should affect other projects. What motivates this question is that the new version of pytest released yesterday raises warnings for non-registered markers, `pytest.mark.slow` in particular, and that was causing CI failures. The easiest way to register a mark is using `pytest.ini`, but we currently don't include that in released wheels, only in source releases. Chuck
On Sun, May 12, 2019 at 3:56 PM Charles R Harris <charlesr.harris@gmail.com> wrote:
On Sun, May 12, 2019 at 6:33 AM Julian Taylor < jtaylor.debian@googlemail.com> wrote:
On 12.05.19 14:58, Charles R Harris wrote:
Hi All,
NumPy currently distinguishes between release and development versions when running tests. Is there a good reason to continue this practice? I ask, because with the last pytest release it would be convenient to always include `pytest.ini ` so that we can register markers. The presence of `pytest.ini` is how we distinguish betweendevelopment from release for testing purposes.
One difference between development and release builds was that in development releases numpy.testing throws errors on floating point exceptions while the release version it did not.
I'd prefer to keep this behavior. It's not clear to me if the proposal is to change the behavior or not. If that is still the case removing the distinction could require a lot
of changes in upstream test suites that are not regularly run against development builds.
The motivation is not quite clear to me, can you please elaborate on what you want to do.
NumPy pytest testing is NumPy specific and not used downstream like our nose testing framework was, so I don't see why that should affect other projects. What motivates this question is that the new version of pytest released yesterday raises warnings for non-registered markers, `pytest.mark.slow` in particular, and that was causing CI failures. The easiest way to register a mark is using `pytest.ini`, but we currently don't include that in released wheels, only in source releases.
Adding a pytest.ini file in wheels should be perfectly fine I think, Ralf
Chuck _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
On Sun, May 12, 2019 at 7:27 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Sun, May 12, 2019 at 3:56 PM Charles R Harris < charlesr.harris@gmail.com> wrote:
On Sun, May 12, 2019 at 6:33 AM Julian Taylor < jtaylor.debian@googlemail.com> wrote:
On 12.05.19 14:58, Charles R Harris wrote:
Hi All,
NumPy currently distinguishes between release and development versions when running tests. Is there a good reason to continue this practice? I ask, because with the last pytest release it would be convenient to always include `pytest.ini ` so that we can register markers. The presence of `pytest.ini` is how we distinguish betweendevelopment from release for testing purposes.
One difference between development and release builds was that in development releases numpy.testing throws errors on floating point exceptions while the release version it did not.
I'd prefer to keep this behavior. It's not clear to me if the proposal is to change the behavior or not.
If that is still the case removing the distinction could require a lot
of changes in upstream test suites that are not regularly run against development builds.
The motivation is not quite clear to me, can you please elaborate on what you want to do.
NumPy pytest testing is NumPy specific and not used downstream like our nose testing framework was, so I don't see why that should affect other projects. What motivates this question is that the new version of pytest released yesterday raises warnings for non-registered markers, `pytest.mark.slow` in particular, and that was causing CI failures. The easiest way to register a mark is using `pytest.ini`, but we currently don't include that in released wheels, only in source releases.
Adding a pytest.ini file in wheels should be perfectly fine I think,
It is the absence of pytest.ini that makes it a release, for that is the file that turns warnings into errors. Chuck
On Sun, May 12, 2019 at 4:41 PM Charles R Harris <charlesr.harris@gmail.com> wrote:
On Sun, May 12, 2019 at 7:27 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Sun, May 12, 2019 at 3:56 PM Charles R Harris < charlesr.harris@gmail.com> wrote:
On Sun, May 12, 2019 at 6:33 AM Julian Taylor < jtaylor.debian@googlemail.com> wrote:
On 12.05.19 14:58, Charles R Harris wrote:
Hi All,
NumPy currently distinguishes between release and development versions when running tests. Is there a good reason to continue this practice? I ask, because with the last pytest release it would be convenient to always include `pytest.ini ` so that we can register markers. The presence of `pytest.ini` is how we distinguish betweendevelopment from release for testing purposes.
One difference between development and release builds was that in development releases numpy.testing throws errors on floating point exceptions while the release version it did not.
I'd prefer to keep this behavior. It's not clear to me if the proposal is to change the behavior or not.
If that is still the case removing the distinction could require a lot
of changes in upstream test suites that are not regularly run against development builds.
The motivation is not quite clear to me, can you please elaborate on what you want to do.
NumPy pytest testing is NumPy specific and not used downstream like our nose testing framework was, so I don't see why that should affect other projects. What motivates this question is that the new version of pytest released yesterday raises warnings for non-registered markers, `pytest.mark.slow` in particular, and that was causing CI failures. The easiest way to register a mark is using `pytest.ini`, but we currently don't include that in released wheels, only in source releases.
Adding a pytest.ini file in wheels should be perfectly fine I think,
It is the absence of pytest.ini that makes it a release, for that is the file that turns warnings into errors.
Why don't we just always keep pytest.ini, and move the settings for warnings-to-errors to runtests.py or tools/travis_test.sh? It's not important how we check, all we need is some mechanism to prevent new warnings creeping in. Same as we set -Wall for building in CI. Cheers, Ralf
On Mon, May 13, 2019 at 3:02 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Sun, May 12, 2019 at 4:41 PM Charles R Harris < charlesr.harris@gmail.com> wrote:
On Sun, May 12, 2019 at 7:27 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Sun, May 12, 2019 at 3:56 PM Charles R Harris < charlesr.harris@gmail.com> wrote:
On Sun, May 12, 2019 at 6:33 AM Julian Taylor < jtaylor.debian@googlemail.com> wrote:
Hi All,
NumPy currently distinguishes between release and development versions when running tests. Is there a good reason to continue this
On 12.05.19 14:58, Charles R Harris wrote: practice? I
ask, because with the last pytest release it would be convenient to always include `pytest.ini ` so that we can register markers. The presence of `pytest.ini` is how we distinguish betweendevelopment from release for testing purposes.
One difference between development and release builds was that in development releases numpy.testing throws errors on floating point exceptions while the release version it did not.
I'd prefer to keep this behavior. It's not clear to me if the proposal is to change the behavior or not.
If that is still the case removing the distinction could require a lot
of changes in upstream test suites that are not regularly run against development builds.
The motivation is not quite clear to me, can you please elaborate on what you want to do.
NumPy pytest testing is NumPy specific and not used downstream like our nose testing framework was, so I don't see why that should affect other projects. What motivates this question is that the new version of pytest released yesterday raises warnings for non-registered markers, `pytest.mark.slow` in particular, and that was causing CI failures. The easiest way to register a mark is using `pytest.ini`, but we currently don't include that in released wheels, only in source releases.
Adding a pytest.ini file in wheels should be perfectly fine I think,
It is the absence of pytest.ini that makes it a release, for that is the file that turns warnings into errors.
Why don't we just always keep pytest.ini, and move the settings for warnings-to-errors to runtests.py or tools/travis_test.sh?
It's not important how we check, all we need is some mechanism to prevent new warnings creeping in. Same as we set -Wall for building in CI.
I just checked that current wheels show the warnings when `numpy.test()` is run with latest pytest. However, moving the `pytest.ini` file into the `numpy` directory is tricky, as we need to tell pytest where to find the installed file (-c option). The simplest short time solution is to ignore the warning, but long term I'm worried that the warning will become an error as pytest is doing this because they want to clean up the their implementation. Ideally there would be a better way to register the marks. Note that we currently deal with the missing `pytest.ini` in wheels by duplicating the warnings filters in `_pytesttester.py` using the pytest command line. Adding the `error` filter in the command line worries me, as I'm not sure how the priorities work out, the command line is now appended to the contents of `pytest.ini`. Chuck
On Mon, May 13, 2019 at 11:25 AM Charles R Harris <charlesr.harris@gmail.com> wrote:
On Mon, May 13, 2019 at 3:02 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Sun, May 12, 2019 at 4:41 PM Charles R Harris < charlesr.harris@gmail.com> wrote:
On Sun, May 12, 2019 at 7:27 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Sun, May 12, 2019 at 3:56 PM Charles R Harris < charlesr.harris@gmail.com> wrote:
On Sun, May 12, 2019 at 6:33 AM Julian Taylor < jtaylor.debian@googlemail.com> wrote:
On 12.05.19 14:58, Charles R Harris wrote: > Hi All, > > NumPy currently distinguishes between release and development versions > when running tests. Is there a good reason to continue this practice? I > ask, because with the last pytest release it would be convenient to > always include `pytest.ini ` so that we can register markers. The > presence of `pytest.ini` is how we distinguish betweendevelopment from > release for testing purposes. >
One difference between development and release builds was that in development releases numpy.testing throws errors on floating point exceptions while the release version it did not.
I'd prefer to keep this behavior. It's not clear to me if the proposal is to change the behavior or not.
If that is still the case removing the distinction could require a lot
of changes in upstream test suites that are not regularly run against development builds.
The motivation is not quite clear to me, can you please elaborate on what you want to do.
NumPy pytest testing is NumPy specific and not used downstream like our nose testing framework was, so I don't see why that should affect other projects. What motivates this question is that the new version of pytest released yesterday raises warnings for non-registered markers, `pytest.mark.slow` in particular, and that was causing CI failures. The easiest way to register a mark is using `pytest.ini`, but we currently don't include that in released wheels, only in source releases.
Adding a pytest.ini file in wheels should be perfectly fine I think,
It is the absence of pytest.ini that makes it a release, for that is the file that turns warnings into errors.
Why don't we just always keep pytest.ini, and move the settings for warnings-to-errors to runtests.py or tools/travis_test.sh?
It's not important how we check, all we need is some mechanism to prevent new warnings creeping in. Same as we set -Wall for building in CI.
I just checked that current wheels show the warnings when `numpy.test()` is run with latest pytest. However, moving the `pytest.ini` file into the `numpy` directory is tricky, as we need to tell pytest where to find the installed file (-c option). The simplest short time solution is to ignore the warning, but long term I'm worried that the warning will become an error as pytest is doing this because they want to clean up the their implementation. Ideally there would be a better way to register the marks. Note that we currently deal with the missing `pytest.ini` in wheels by duplicating the warnings filters in `_pytesttester.py` using the pytest command line.
Adding the `error` filter in the command line worries me, as I'm not sure how the priorities work out, the command line is now appended to the contents of `pytest.ini`.
Note that this is also a problem when numpy is installed with `setup.py install`. Things work now for runtests because `pytest.ini` is in the directory from which the tests are run. Chuck
On Mon, May 13, 2019, 10:26 Charles R Harris <charlesr.harris@gmail.com> wrote:
I just checked that current wheels show the warnings when `numpy.test()` is run with latest pytest. However, moving the `pytest.ini` file into the `numpy` directory is tricky, as we need to tell pytest where to find the installed file (-c option). The simplest short time solution is to ignore the warning, but long term I'm worried that the warning will become an error as pytest is doing this because they want to clean up the their implementation. Ideally there would be a better way to register the marks.
I bet if you open an issue on pytest explaining that numpy needs to either use unregistered marks or else have some programmatic non-pytest.ini-based way to register marks, then they'll figure something out. I think they added a warning because they're hoping to flush out these kinds of problems so they can fix them before they do the cleanup. -n
On Mon, May 13, 2019 at 11:50 AM Nathaniel Smith <njs@pobox.com> wrote:
On Mon, May 13, 2019, 10:26 Charles R Harris <charlesr.harris@gmail.com> wrote:
I just checked that current wheels show the warnings when `numpy.test()` is run with latest pytest. However, moving the `pytest.ini` file into the `numpy` directory is tricky, as we need to tell pytest where to find the installed file (-c option). The simplest short time solution is to ignore the warning, but long term I'm worried that the warning will become an error as pytest is doing this because they want to clean up the their implementation. Ideally there would be a better way to register the marks.
I bet if you open an issue on pytest explaining that numpy needs to either use unregistered marks or else have some programmatic non-pytest.ini-based way to register marks, then they'll figure something out. I think they added a warning because they're hoping to flush out these kinds of problems so they can fix them before they do the cleanup.
Turns out that markers can be registered in conftest.py, so that fixes the problem for NumPy. SciPy will need this fix also. Chuck
participants (4)
-
Charles R Harris -
Julian Taylor -
Nathaniel Smith -
Ralf Gommers