[Twisted-Python] Twisted trunk and klein?
Hi, I quickly asked Wilfredo if Twisted trunk worked with Klein, and he mentioned that he saw this dependency issue on the Twisted klein side when used with Twisted trunk: The conflict is caused by: The user requested idna==3.1 hyperlink 21.0.0 depends on idna>=2.5 requests 2.25.1 depends on idna<3 and >=2.5 Is that problem purely for klein to deal with, or is there any issue that should be fixed in Twisted trunk? -- Craig
On Feb 10, 2021, at 3:18 PM, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
Hi,
I quickly asked Wilfredo if Twisted trunk worked with Klein, and he mentioned that he saw this dependency issue on the Twisted klein side when used with Twisted trunk:
The conflict is caused by: The user requested idna==3.1 hyperlink 21.0.0 depends on idna>=2.5 requests 2.25.1 depends on idna<3 and >=2.5
Is that problem purely for klein to deal with, or is there any issue that should be fixed in Twisted trunk?
This isn't even a bug in Klein, it's an issue with a version pin in its tox.ini: https://github.com/twisted/klein/blob/6e7b37158dea2fe73180809803a872ed98143c... <https://github.com/twisted/klein/blob/6e7b37158dea2fe73180809803a872ed98143c...> The constraints from requests (<3,>=2.5) and hyperlink (>=2.5) are perfectly compatible; one's just a subset of the other. So it might be nice to submit a Klein PR at some point soon, but it's not a release blocker. -g
On Feb 10, 2021, at 3:44 PM, Glyph <glyph@twistedmatrix.com> wrote:
This isn't even a bug in Klein, it's an issue with a version pin in its tox.ini: https://github.com/twisted/klein/blob/6e7b37158dea2fe73180809803a872ed98143c... <https://github.com/twisted/klein/blob/6e7b37158dea2fe73180809803a872ed98143c...>
The constraints from requests (<3,>=2.5) and hyperlink (>=2.5) are perfectly compatible; one's just a subset of the other.
Sure, they are compatible, but Klein doesn’t use requests directly, and this only looks to be failing in trunk. My concern here is that Twisted may have added a dependency on requests, and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini does, that you aren’t allowed to. That seems weak; I’d say a bug. No? -wsv
My concern here is that Twisted may have added a dependency on requests, and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini does, that you aren’t allowed to. That seems weak; I’d say a bug. No?
Note this is only cause when installing treq, so the issue seems to be there, though for some reason, it’s only a problem with Twisted trunk, which kind of confuses me, but I haven’t dug into it beyond trying to figure out a bit about what fails to install. So I agree probably it’s not a blocker for Twisted, and it’s only annoying to Klein for its own testing, not it’s clients, but if I were a client of treq (and Klein is), I’d think this is a bug. -wsv
On Feb 11, 2021, at 11:41 AM, Wilfredo Sánchez Vega <wsanchez@wsanchez.net> wrote:
My concern here is that Twisted may have added a dependency on requests, and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini does, that you aren’t allowed to. That seems weak; I’d say a bug. No?
Note this is only cause when installing treq, so the issue seems to be there, though for some reason, it’s only a problem with Twisted trunk, which kind of confuses me, but I haven’t dug into it beyond trying to figure out a bit about what fails to install.
So I agree probably it’s not a blocker for Twisted, and it’s only annoying to Klein for its own testing, not it’s clients, but if I were a client of treq (and Klein is), I’d think this is a bug.
Attempting to reproduce this locally was a real adventure for me. I had a bunch of failure-to-terminate cases from the new pip resolver, until I gave up, download the newest security update of each release of python, upgraded pip, tox, setuptools, and virtualenv for every python version, fully cleaned my git checkout, and re-ran. Now everything passes except for coverage-py39-twtrunk, which still fails to terminate. Under -vvvv, it spends a very long time printing messages like these over and over again: INFO: pip is looking at multiple versions of setuptools to determine which version is compatible with other requirements. This could take a while. INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking <https://pip.pypa.io/surveys/backtracking> as it installs every version of setuptools and six that has ever existed. It seems like Klein has backed itself into a very weird and complex corner with the new dependency resolver, but it doesn't appear unique to Twisted trunk, just perhaps tickled ever so slightly worse. -g
On Feb 11, 2021, at 1:04 PM, Glyph <glyph@twistedmatrix.com> wrote:
On Feb 11, 2021, at 11:41 AM, Wilfredo Sánchez Vega <wsanchez@wsanchez.net <mailto:wsanchez@wsanchez.net>> wrote:
My concern here is that Twisted may have added a dependency on requests, and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini does, that you aren’t allowed to. That seems weak; I’d say a bug. No?
Note this is only cause when installing treq, so the issue seems to be there, though for some reason, it’s only a problem with Twisted trunk, which kind of confuses me, but I haven’t dug into it beyond trying to figure out a bit about what fails to install.
So I agree probably it’s not a blocker for Twisted, and it’s only annoying to Klein for its own testing, not it’s clients, but if I were a client of treq (and Klein is), I’d think this is a bug.
Attempting to reproduce this locally was a real adventure for me. I had a bunch of failure-to-terminate cases from the new pip resolver, until I gave up, download the newest security update of each release of python, upgraded pip, tox, setuptools, and virtualenv for every python version, fully cleaned my git checkout, and re-ran.
Now everything passes except for coverage-py39-twtrunk, which still fails to terminate.
Under -vvvv, it spends a very long time printing messages like these over and over again:
INFO: pip is looking at multiple versions of setuptools to determine which version is compatible with other requirements. This could take a while. INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking <https://pip.pypa.io/surveys/backtracking>
as it installs every version of setuptools and six that has ever existed.
It seems like Klein has backed itself into a very weird and complex corner with the new dependency resolver, but it doesn't appear unique to Twisted trunk, just perhaps tickled ever so slightly worse.
I've now been running `tox -vvvv -r -e coverage-py39-twtrunk` for well over an hour, so I think we may have some pip resolver bugs to report. -g
On Feb 11, 2021, at 2:09 PM, Glyph <glyph@twistedmatrix.com> wrote:
On Feb 11, 2021, at 1:04 PM, Glyph <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
On Feb 11, 2021, at 11:41 AM, Wilfredo Sánchez Vega <wsanchez@wsanchez.net <mailto:wsanchez@wsanchez.net>> wrote:
My concern here is that Twisted may have added a dependency on requests, and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini does, that you aren’t allowed to. That seems weak; I’d say a bug. No?
Note this is only cause when installing treq, so the issue seems to be there, though for some reason, it’s only a problem with Twisted trunk, which kind of confuses me, but I haven’t dug into it beyond trying to figure out a bit about what fails to install.
So I agree probably it’s not a blocker for Twisted, and it’s only annoying to Klein for its own testing, not it’s clients, but if I were a client of treq (and Klein is), I’d think this is a bug.
Attempting to reproduce this locally was a real adventure for me. I had a bunch of failure-to-terminate cases from the new pip resolver, until I gave up, download the newest security update of each release of python, upgraded pip, tox, setuptools, and virtualenv for every python version, fully cleaned my git checkout, and re-ran.
Now everything passes except for coverage-py39-twtrunk, which still fails to terminate.
Under -vvvv, it spends a very long time printing messages like these over and over again:
INFO: pip is looking at multiple versions of setuptools to determine which version is compatible with other requirements. This could take a while. INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking <https://pip.pypa.io/surveys/backtracking>
as it installs every version of setuptools and six that has ever existed.
It seems like Klein has backed itself into a very weird and complex corner with the new dependency resolver, but it doesn't appear unique to Twisted trunk, just perhaps tickled ever so slightly worse.
I've now been running `tox -vvvv -r -e coverage-py39-twtrunk` for well over an hour, so I think we may have some pip resolver bugs to report.
I filed https://github.com/pypa/pip/issues/9601 <https://github.com/pypa/pip/issues/9601> for this since it seems like it's well outside of our control. If anyone else can add more information that would be great. -g
On Thu, Feb 11, 2021 at 2:29 PM Glyph <glyph@twistedmatrix.com> wrote:
I filed https://github.com/pypa/pip/issues/9601 for this since it seems like it's well outside of our control. If anyone else can add more information that would be great.
Thanks for doing the analysis and filing that bug. I didn't want to jump on anything, and wanted to give some breathing room for folks to look at that and chime in. Wilfredo has this PR to klein where he is also looking at these issues: https://github.com/twisted/klein/pull/450 So I think that I will not hold up the Twisted core release based on this issue. Hopefully this issue can be resolved outside of Twisted core. -- Craig
On Thu, 11 Feb 2021 at 19:26, Wilfredo Sánchez Vega <wsanchez@wsanchez.net> wrote:
On Feb 10, 2021, at 3:44 PM, Glyph <glyph@twistedmatrix.com> wrote:
This isn't even a bug in Klein, it's an issue with a version pin in its tox.ini: https://github.com/twisted/klein/blob/6e7b37158dea2fe73180809803a872ed98143c...
The constraints from requests (<3,>=2.5) and hyperlink (>=2.5) are perfectly compatible; one's just a subset of the other.
Sure, they are compatible, but Klein doesn’t use requests directly, and this only looks to be failing in trunk.
My concern here is that Twisted may have added a dependency on requests, and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini does, that you aren’t allowed to. That seems weak; I’d say a bug. No?
I have created a new virtual env and executed `pip install .` on trunk. It installed the following packages: constantly, zope.interface, attrs, incremental, six, Automat, idna, hyperlink, Twisted ------------ `requests` was not installed. I wasn't expecting to see `idna` as it is listed as TLS only deps.... but it looks like hyperlink depends on idna. Here is the dep tree $ pipdeptree -fl Twisted @ file:///home/adi/dev/twisted attrs==20.3.0 Automat==20.2.0 attrs==20.3.0 six==1.15.0 constantly==15.1.0 hyperlink==21.0.0 idna==3.1 incremental==17.5.0 zope.interface==5.2.0 setuptools==44.0.0 wheel==0.36.2 ------------ I guess that we can remove idna from setup.cfg TLS section in Twisted tls = pyopenssl >= 16.0.0 # service_identity 18.1.0 added support for validating IP addresses in # certificate subjectAltNames service_identity >= 18.1.0 idna >= 2.4 Cheers -- Adi Roiban
On Feb 11, 2021, at 1:15 PM, Adi Roiban <adi@roiban.ro> wrote:
I guess that we can remove idna from setup.cfg TLS section in Twisted
Nope; we use it directly, in https://github.com/twisted/twisted/blob/7cf6c8bc320ac5fd96b4784f6feb932ea819... <https://github.com/twisted/twisted/blob/7cf6c8bc320ac5fd96b4784f6feb932ea819...>. In fact, if anything, we should bump it up, since we use it even in non-tls configurations: https://github.com/twisted/twisted/blob/cd97222df2ca7032bbff2fe9a8793d7b42de... <https://github.com/twisted/twisted/blob/cd97222df2ca7032bbff2fe9a8793d7b42de...> . -g
participants (4)
-
Adi Roiban -
Craig Rodrigues -
Glyph -
Wilfredo Sánchez Vega