I've recently updated my tox install from 3.0.0 to 3.1.1. There's a new warning: /home/fdrake/tools/tox/lib/python3.6/site-packages/tox/config.py:527: UserWarning: Conflicting basepython for environment 'py36'; resolve conflict or configure ignore_basepython_conflict testenv_config.envname, str(value), default My configuration includes basepython settings for what are called "default factors" in the updated documentation (I couldn't find the term in the 3.0.0 docs): [testenv:py27] basepython = /opt/kt-python27/bin/python2 [testenv:py35] basepython = /opt/kt-python35/bin/python3 [testenv:py36] basepython = /opt/kt-python36/bin/python3 [testenv:py37] basepython = /opt/kt-python37/bin/python3 In my environment, there are no "system" Python installations for Python 3.6 or 3.7. I see the docs for the setting "ignore_basepython_conflict", and would like to understand how tox is intended to be configured for alternate Python installations like mine. There's a warning in the doc that the warning will become an error in the future; will that be the case even when "ignore_basepython_conflict" is true? Is it intended that using alternate Python installations requires the definition of environments that do not include the "default factors"? I.e., should I be defining "ktpy35", "ktpy36", etc., and only using those? Thanks! -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein
On Thu, Jul 12, 2018 at 10:23 AM Fred Drake <fred@fdrake.net> wrote:
I see the docs for the setting "ignore_basepython_conflict", and would like to understand how tox is intended to be configured for alternate Python installations like mine.
Playing further with it, setting "ignore_basepython_conflict = true" causes my setting of the value to be ignored, so the Python interpreter for py36 and py37 aren't found at all. -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein
Hi Fred, this is a fix introduced new in 3.1 to make sure that the wrong interpreter isn't used silently when tox is misconfigured. I wasn't involved in the fix and lack the details, but your config looks like there should be no such warning. When you say that py36 and py37 weren't found at all: do you mean that the warning shadows an actual error? If this is the case, it would be great if you could open an issue with a quick explanation how to reproduce this. Here are the details for this issue and the changes, if you want to look further into this: https://github.com/tox-dev/tox/commit/d5b9c0a93931707b55f8f64bbd37be7d0b6a20... https://github.com/tox-dev/tox/issues/477 Cheers, Oliver On Thu, 12 Jul 2018 at 16:39 Fred Drake <fred@fdrake.net> wrote:
On Thu, Jul 12, 2018 at 10:23 AM Fred Drake <fred@fdrake.net> wrote:
I see the docs for the setting "ignore_basepython_conflict", and would like to understand how tox is intended to be configured for alternate Python installations like mine.
Playing further with it, setting "ignore_basepython_conflict = true" causes my setting of the value to be ignored, so the Python interpreter for py36 and py37 aren't found at all.
-Fred
-- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein _______________________________________________ tox-dev mailing list -- tox-dev@python.org To unsubscribe send an email to tox-dev-leave@python.org https://mail.python.org/mm3/mailman3/lists/tox-dev.python.org/
This is working at intended. tox is about reproducible build, hard coding to full path seems from this point wrong as it's almost bound to fail on other peoples system. Discovery on Linux systems should be via having the pythonx.y format as specifiad in http://www.python.org/dev/peps/pep-0394/#application-to-the-cpython-referenc.... tox follows this logic for interpreter discovery, especially for pre-defined build environments. That being said I know this worked until now. So in that sense it's a regression. I would say though that your use case is the edge case, if you want to use it like that define custom tox environments, e.g. py27k instead of py27. In general, for people who follow the python discovery system enlisted in that pep, this change should help them avoid invalid configuration and bad suprises. On Jul 13, 2018 10:55, "Oliver Bestwalter" <oliver@bestwalter.de> wrote: Hi Fred, this is a fix introduced new in 3.1 to make sure that the wrong interpreter isn't used silently when tox is misconfigured. I wasn't involved in the fix and lack the details, but your config looks like there should be no such warning. When you say that py36 and py37 weren't found at all: do you mean that the warning shadows an actual error? If this is the case, it would be great if you could open an issue with a quick explanation how to reproduce this. Here are the details for this issue and the changes, if you want to look further into this: https://github.com/tox-dev/tox/commit/d5b9c0a93931707b55f8f64bbd37be7d0b6a20... https://github.com/tox-dev/tox/issues/477 Cheers, Oliver On Thu, 12 Jul 2018 at 16:39 Fred Drake <fred@fdrake.net> wrote:
On Thu, Jul 12, 2018 at 10:23 AM Fred Drake <fred@fdrake.net> wrote:
I see the docs for the setting "ignore_basepython_conflict", and would like to understand how tox is intended to be configured for alternate Python installations like mine.
Playing further with it, setting "ignore_basepython_conflict = true" causes my setting of the value to be ignored, so the Python interpreter for py36 and py37 aren't found at all.
-Fred
-- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein _______________________________________________ tox-dev mailing list -- tox-dev@python.org To unsubscribe send an email to tox-dev-leave@python.org https://mail.python.org/mm3/mailman3/lists/tox-dev.python.org/
_______________________________________________ tox-dev mailing list -- tox-dev@python.org To unsubscribe send an email to tox-dev-leave@python.org https://mail.python.org/mm3/mailman3/lists/tox-dev.python.org/
On Fri, Jul 13, 2018 at 5:55 AM Oliver Bestwalter <oliver@bestwalter.de> wrote:
this is a fix introduced new in 3.1 to make sure that the wrong interpreter isn't used silently when tox is misconfigured. I wasn't involved in the fix and lack the details, but your config looks like there should be no such warning.
With the attached tox.ini and Python interpreters at the indicated locations, I see four warnings, but the configured interpreters are used: py35 recreate: /home/fdrake/p/kt.testing/.tox/py35 /home/fdrake/p/kt.testing/.tox$ /home/fdrake/tools/tox/bin/python3 -m virtualenv --python /opt/kt-python35/bin/python3 py35
/home/fdrake/p/kt.testing/.tox/py35/log/py35-0.log
Uncommenting the "ignore_basepython_conflict = true" setting, the configured locations of the interpreters is ignored, and no warnings are displayed: py35 recreate: /home/fdrake/p/kt.testing/.tox/py35 /home/fdrake/p/kt.testing/.tox$ /home/fdrake/tools/tox/bin/python3 -m virtualenv --python /usr/bin/python3.5 py35
/home/fdrake/p/kt.testing/.tox/py35/log/py35-0.log
When you say that py36 and py37 weren't found at all: do you mean that the warning shadows an actual error? If this is the case, it would be great if you could open an issue with a quick explanation how to reproduce this.
With "ignore_basepython_conflict = true" uncommented, tox ignores the basepython settings and tells me the interpreters are not found. Since I've used "skip_missing_interpreters = true", it skips without an error: py36 recreate: /home/fdrake/p/kt.testing/.tox/py36 SKIPPED: InterpreterNotFound: python3.6 py37 recreate: /home/fdrake/p/kt.testing/.tox/py37 SKIPPED: InterpreterNotFound: python3.7 Using "ignore_basepython_conflict = true" and "skip_missing_interpreters = false", the system 2.7 and 3.5 are used, and 3.6 and 3.7 are reported missing as errors: py36 recreate: /home/fdrake/p/kt.testing/.tox/py36 ERROR: InterpreterNotFound: python3.6 py37 recreate: /home/fdrake/p/kt.testing/.tox/py37 ERROR: InterpreterNotFound: python3.7 So I think there are a couple of surprising behaviors here. 1. A warning about a conflict is being generated when the Python I've specified actually matches the version I've indicated I want (by factor). The interpreter isn't from a system package (provided by the OS distribution, Ubuntu in my case), but for each of my configured interpreters, the version matches. 2. When the configuration indicates the warning should be suppressed, the configuration of the basepython setting is ignored. This strikes me as counter-intuitive. Warning of suspicious configuration is arguably a good idea, but ignoring configuration is never the right thing to do. If the user agrees with the warning, they'll change the configuration and have another go at it. It remains unclear to be whether the "default factors" are intended to refer to specific versions of Python, or to specific installations. Searching for defaults based on the PATH is reasonable, but many software engineers will also argue that building apps on top of them is insane. When testing a library with tox, I want to use the same Python's I'll consider for my applications. -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein
That being said it would be possible to add a smarter check against python version missmatch that would still allow this kind of path hardcoding. On Fri, Jul 13, 2018, 15:21 Fred Drake <fred@fdrake.net> wrote:
On Fri, Jul 13, 2018 at 5:55 AM Oliver Bestwalter <oliver@bestwalter.de> wrote:
this is a fix introduced new in 3.1 to make sure that the wrong interpreter isn't used silently when tox is misconfigured. I wasn't involved in the fix and lack the details, but your config looks like there should be no such warning.
With the attached tox.ini and Python interpreters at the indicated locations, I see four warnings, but the configured interpreters are used:
py35 recreate: /home/fdrake/p/kt.testing/.tox/py35 /home/fdrake/p/kt.testing/.tox$ /home/fdrake/tools/tox/bin/python3 -m virtualenv --python /opt/kt-python35/bin/python3 py35
/home/fdrake/p/kt.testing/.tox/py35/log/py35-0.log
Uncommenting the "ignore_basepython_conflict = true" setting, the configured locations of the interpreters is ignored, and no warnings are displayed:
py35 recreate: /home/fdrake/p/kt.testing/.tox/py35 /home/fdrake/p/kt.testing/.tox$ /home/fdrake/tools/tox/bin/python3 -m virtualenv --python /usr/bin/python3.5 py35
/home/fdrake/p/kt.testing/.tox/py35/log/py35-0.log
When you say that py36 and py37 weren't found at all: do you mean that the warning shadows an actual error? If this is the case, it would be great if you could open an issue with a quick explanation how to reproduce this.
With "ignore_basepython_conflict = true" uncommented, tox ignores the basepython settings and tells me the interpreters are not found. Since I've used "skip_missing_interpreters = true", it skips without an error:
py36 recreate: /home/fdrake/p/kt.testing/.tox/py36 SKIPPED: InterpreterNotFound: python3.6 py37 recreate: /home/fdrake/p/kt.testing/.tox/py37 SKIPPED: InterpreterNotFound: python3.7
Using "ignore_basepython_conflict = true" and "skip_missing_interpreters = false", the system 2.7 and 3.5 are used, and 3.6 and 3.7 are reported missing as errors:
py36 recreate: /home/fdrake/p/kt.testing/.tox/py36 ERROR: InterpreterNotFound: python3.6 py37 recreate: /home/fdrake/p/kt.testing/.tox/py37 ERROR: InterpreterNotFound: python3.7
So I think there are a couple of surprising behaviors here.
1. A warning about a conflict is being generated when the Python I've specified actually matches the version I've indicated I want (by factor). The interpreter isn't from a system package (provided by the OS distribution, Ubuntu in my case), but for each of my configured interpreters, the version matches.
2. When the configuration indicates the warning should be suppressed, the configuration of the basepython setting is ignored. This strikes me as counter-intuitive. Warning of suspicious configuration is arguably a good idea, but ignoring configuration is never the right thing to do. If the user agrees with the warning, they'll change the configuration and have another go at it.
It remains unclear to be whether the "default factors" are intended to refer to specific versions of Python, or to specific installations. Searching for defaults based on the PATH is reasonable, but many software engineers will also argue that building apps on top of them is insane. When testing a library with tox, I want to use the same Python's I'll consider for my applications.
-Fred
-- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein _______________________________________________ tox-dev mailing list -- tox-dev@python.org To unsubscribe send an email to tox-dev-leave@python.org https://mail.python.org/mm3/mailman3/lists/tox-dev.python.org/
On Fri, Jul 13, 2018 at 10:16 AM Bernát Gábor <jokerjokerer@gmail.com> wrote:
This is working at intended. tox is about reproducible build, hard coding to full path seems from this point wrong as it's almost bound to fail on other peoples system. Discovery on Linux systems should be via having the pythonx.y format as specifiad in http://www.python.org/dev/peps/pep-0394/#application-to-the-cpython-referenc.... tox follows this logic for interpreter discovery, especially for pre-defined build environments.
That PEP deals with how symlinks should be managed for interpreters accessed via PATH; it really doesn't say anything about selecting the Python to use for a particular application. In the end, it always boils down to "run the Python you want"; there's no reason to expect all interpreters to be installed along the PATH.
That being said I know this worked until now. So in that sense it's a regression. I would say though that your use case is the edge case, if you want to use it like that define custom tox environments, e.g. py27k instead of py27. In general, for people who follow the python discovery system enlisted in that pep, this change should help them avoid invalid configuration and bad suprises.
Issuing a warning when the Python version doesn't match a pyXY factor, if used, makes sense; I agree that's a valuable thing and can let people know when they're shooting themselves in the foot. But the warning is the right reaction to that; I don't think ignoring the configuration is ever the right thing to do. In the package this example is taken from (open source), I'm sharing the tox.ini except for the basepython settings: https://github.com/keepertech/kt.testing/blob/master/tox.ini I'd love to have a way to create a second configuration (kt-tox.ini) that includes the tox.ini and adds the basepython settings. That would allow me to avoid needing to repeat the rest of the configuration in a second file. (Buildout supports this with a [buildout] extends = ... setting; a [tox] extends = ... would be really nice to have for this.) On Fri, Jul 13, 2018 at 10:38 AM Bernát Gábor <jokerjokerer@gmail.com> wrote:
That being said it would be possible to add a smarter check against python version missmatch that would still allow this kind of path hardcoding.
If the intent is to verify & warn the user that an environment including a pyXY factor is using some Python version other than X.Y, the check should be simple; just check the return code from: {envdir}/bin/python -c 'import sys; sys.exit(sys.version_info[:2] != (X, Y))' I think this would be a better check. -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein
Yes, the later is the intent and indeed what you're suggesting is what would be more robust solution. Please create an issue on GitHub for it and we'll release a patch fix. On Fri, Jul 13, 2018, 16:22 Fred Drake <fred@fdrake.net> wrote:
This is working at intended. tox is about reproducible build, hard coding to full path seems from this point wrong as it's almost bound to fail on other peoples system. Discovery on Linux systems should be via having the pythonx.y format as specifiad in http://www.python.org/dev/peps/pep-0394/#application-to-the-cpython-referenc.... tox follows this logic for interpreter discovery, especially for
On Fri, Jul 13, 2018 at 10:16 AM Bernát Gábor <jokerjokerer@gmail.com> wrote: pre-defined build environments.
That PEP deals with how symlinks should be managed for interpreters accessed via PATH; it really doesn't say anything about selecting the Python to use for a particular application. In the end, it always boils down to "run the Python you want"; there's no reason to expect all interpreters to be installed along the PATH.
That being said I know this worked until now. So in that sense it's a regression. I would say though that your use case is the edge case, if you want to use it like that define custom tox environments, e.g. py27k instead of py27. In general, for people who follow the python discovery system enlisted in that pep, this change should help them avoid invalid configuration and bad suprises.
Issuing a warning when the Python version doesn't match a pyXY factor, if used, makes sense; I agree that's a valuable thing and can let people know when they're shooting themselves in the foot. But the warning is the right reaction to that; I don't think ignoring the configuration is ever the right thing to do.
In the package this example is taken from (open source), I'm sharing the tox.ini except for the basepython settings:
https://github.com/keepertech/kt.testing/blob/master/tox.ini
I'd love to have a way to create a second configuration (kt-tox.ini) that includes the tox.ini and adds the basepython settings. That would allow me to avoid needing to repeat the rest of the configuration in a second file. (Buildout supports this with a [buildout] extends = ... setting; a [tox] extends = ... would be really nice to have for this.)
That being said it would be possible to add a smarter check against
On Fri, Jul 13, 2018 at 10:38 AM Bernát Gábor <jokerjokerer@gmail.com> wrote: python version missmatch that would still allow this kind of path hardcoding.
If the intent is to verify & warn the user that an environment including a pyXY factor is using some Python version other than X.Y, the check should be simple; just check the return code from:
{envdir}/bin/python -c 'import sys; sys.exit(sys.version_info[:2] != (X, Y))'
I think this would be a better check.
-Fred
-- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein
On Fri, Jul 13, 2018 at 11:28 AM Bernát Gábor <jokerjokerer@gmail.com> wrote:
Yes, the later is the intent and indeed what you're suggesting is what would be more robust solution. Please create an issue on GitHub for it and we'll release a patch fix.
https://github.com/tox-dev/tox/issues/908 -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein
participants (3)
-
Bernát Gábor -
Fred Drake -
Oliver Bestwalter