Mailman 3 [Twisted-Python] Twisted trunk and klein? - Twisted - python.org

newer
[Twisted-Python] Weekly Bug Summary

[Twisted-Python] Twisted trunk and klein?

older
[Twisted-Python] Moving...

Craig Rodrigues

11 Feb 2021 11 Feb '21

4:48 a.m.

Hi, I quickly asked Wilfredo if Twisted trunk worked with Klein, and he mentioned that he saw this dependency issue on the Twisted klein side when used with Twisted trunk: The conflict is caused by: The user requested idna==3.1 hyperlink 21.0.0 depends on idna>=2.5 requests 2.25.1 depends on idna<3 and >=2.5 Is that problem purely for klein to deal with, or is there any issue that should be fixed in Twisted trunk? -- Craig

Reply

Sign in to reply online Use email software

Show replies by date

Glyph

11 Feb 11 Feb

5:14 a.m.

On Feb 10, 2021, at 3:18 PM, Craig Rodrigues wrote:

Hi,

I quickly asked Wilfredo if Twisted trunk worked with Klein, and he mentioned that he saw this dependency issue on the Twisted klein side when used with Twisted trunk:

The conflict is caused by: The user requested idna==3.1 hyperlink 21.0.0 depends on idna>=2.5 requests 2.25.1 depends on idna<3 and >=2.5

Is that problem purely for klein to deal with, or is there any issue that should be fixed in Twisted trunk?

This isn't even a bug in Klein, it's an issue with a version pin in its tox.ini: https://github.com/twisted/klein/blob/6e7b37158dea2fe73180809803a872ed98143c... https://github.com/twisted/klein/blob/6e7b37158dea2fe73180809803a872ed98143c... The constraints from requests (<3,>=2.5) and hyperlink (>=2.5) are perfectly compatible; one's just a subset of the other. So it might be nice to submit a Klein PR at some point soon, but it's not a release blocker. -g

Reply

Sign in to reply online Use email software

Wilfredo Sánchez Vega

12 Feb 12 Feb

12:55 a.m.

On Feb 10, 2021, at 3:44 PM, Glyph wrote:

This isn't even a bug in Klein, it's an issue with a version pin in its tox.ini: https://github.com/twisted/klein/blob/6e7b37158dea2fe73180809803a872ed98143c... https://github.com/twisted/klein/blob/6e7b37158dea2fe73180809803a872ed98143c...

The constraints from requests (<3,>=2.5) and hyperlink (>=2.5) are perfectly compatible; one's just a subset of the other.

Sure, they are compatible, but Klein doesn’t use requests directly, and this only looks to be failing in trunk. My concern here is that Twisted may have added a dependency on requests, and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini does, that you aren’t allowed to. That seems weak; I’d say a bug. No? -wsv

Reply

Sign in to reply online Use email software

Wilfredo Sánchez Vega

1:11 a.m.

My concern here is that Twisted may have added a dependency on requests, and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini does, that you aren’t allowed to. That seems weak; I’d say a bug. No?

Note this is only cause when installing treq, so the issue seems to be there, though for some reason, it’s only a problem with Twisted trunk, which kind of confuses me, but I haven’t dug into it beyond trying to figure out a bit about what fails to install. So I agree probably it’s not a blocker for Twisted, and it’s only annoying to Klein for its own testing, not it’s clients, but if I were a client of treq (and Klein is), I’d think this is a bug. -wsv

Reply

Sign in to reply online Use email software

Glyph

2:34 a.m.

On Feb 11, 2021, at 11:41 AM, Wilfredo Sánchez Vega wrote:

...
My concern here is that Twisted may have added a dependency on requests, and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini does, that you aren’t allowed to. That seems weak; I’d say a bug. No?

Note this is only cause when installing treq, so the issue seems to be there, though for some reason, it’s only a problem with Twisted trunk, which kind of confuses me, but I haven’t dug into it beyond trying to figure out a bit about what fails to install.

So I agree probably it’s not a blocker for Twisted, and it’s only annoying to Klein for its own testing, not it’s clients, but if I were a client of treq (and Klein is), I’d think this is a bug.

Attempting to reproduce this locally was a real adventure for me. I had a bunch of failure-to-terminate cases from the new pip resolver, until I gave up, download the newest security update of each release of python, upgraded pip, tox, setuptools, and virtualenv for every python version, fully cleaned my git checkout, and re-ran. Now everything passes except for coverage-py39-twtrunk, which still fails to terminate. Under -vvvv, it spends a very long time printing messages like these over and over again: INFO: pip is looking at multiple versions of setuptools to determine which version is compatible with other requirements. This could take a while. INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking https://pip.pypa.io/surveys/backtracking as it installs every version of setuptools and six that has ever existed. It seems like Klein has backed itself into a very weird and complex corner with the new dependency resolver, but it doesn't appear unique to Twisted trunk, just perhaps tickled ever so slightly worse. -g

Reply

Sign in to reply online Use email software

Glyph

3:39 a.m.

On Feb 11, 2021, at 1:04 PM, Glyph wrote:

...
On Feb 11, 2021, at 11:41 AM, Wilfredo Sánchez Vega mailto:wsanchez@wsanchez.net> wrote:

...
My concern here is that Twisted may have added a dependency on requests, and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini does, that you aren’t allowed to. That seems weak; I’d say a bug. No?

Note this is only cause when installing treq, so the issue seems to be there, though for some reason, it’s only a problem with Twisted trunk, which kind of confuses me, but I haven’t dug into it beyond trying to figure out a bit about what fails to install.

So I agree probably it’s not a blocker for Twisted, and it’s only annoying to Klein for its own testing, not it’s clients, but if I were a client of treq (and Klein is), I’d think this is a bug.

Attempting to reproduce this locally was a real adventure for me. I had a bunch of failure-to-terminate cases from the new pip resolver, until I gave up, download the newest security update of each release of python, upgraded pip, tox, setuptools, and virtualenv for every python version, fully cleaned my git checkout, and re-ran.

Now everything passes except for coverage-py39-twtrunk, which still fails to terminate.

Under -vvvv, it spends a very long time printing messages like these over and over again:

INFO: pip is looking at multiple versions of setuptools to determine which version is compatible with other requirements. This could take a while. INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking https://pip.pypa.io/surveys/backtracking

as it installs every version of setuptools and six that has ever existed.

It seems like Klein has backed itself into a very weird and complex corner with the new dependency resolver, but it doesn't appear unique to Twisted trunk, just perhaps tickled ever so slightly worse.

I've now been running `tox -vvvv -r -e coverage-py39-twtrunk` for well over an hour, so I think we may have some pip resolver bugs to report. -g

Reply

Sign in to reply online Use email software

Glyph

3:59 a.m.

On Feb 11, 2021, at 2:09 PM, Glyph wrote:

...
On Feb 11, 2021, at 1:04 PM, Glyph mailto:glyph@twistedmatrix.com> wrote:

...
On Feb 11, 2021, at 11:41 AM, Wilfredo Sánchez Vega mailto:wsanchez@wsanchez.net> wrote:

...
My concern here is that Twisted may have added a dependency on requests, and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini does, that you aren’t allowed to. That seems weak; I’d say a bug. No?

Note this is only cause when installing treq, so the issue seems to be there, though for some reason, it’s only a problem with Twisted trunk, which kind of confuses me, but I haven’t dug into it beyond trying to figure out a bit about what fails to install.

So I agree probably it’s not a blocker for Twisted, and it’s only annoying to Klein for its own testing, not it’s clients, but if I were a client of treq (and Klein is), I’d think this is a bug.

Attempting to reproduce this locally was a real adventure for me. I had a bunch of failure-to-terminate cases from the new pip resolver, until I gave up, download the newest security update of each release of python, upgraded pip, tox, setuptools, and virtualenv for every python version, fully cleaned my git checkout, and re-ran.

Now everything passes except for coverage-py39-twtrunk, which still fails to terminate.

Under -vvvv, it spends a very long time printing messages like these over and over again:

INFO: pip is looking at multiple versions of setuptools to determine which version is compatible with other requirements. This could take a while. INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking https://pip.pypa.io/surveys/backtracking

as it installs every version of setuptools and six that has ever existed.

It seems like Klein has backed itself into a very weird and complex corner with the new dependency resolver, but it doesn't appear unique to Twisted trunk, just perhaps tickled ever so slightly worse.

I've now been running `tox -vvvv -r -e coverage-py39-twtrunk` for well over an hour, so I think we may have some pip resolver bugs to report.

I filed https://github.com/pypa/pip/issues/9601 https://github.com/pypa/pip/issues/9601 for this since it seems like it's well outside of our control. If anyone else can add more information that would be great. -g

Reply

Sign in to reply online Use email software

Craig Rodrigues

13 Feb 13 Feb

7:27 a.m.

On Thu, Feb 11, 2021 at 2:29 PM Glyph wrote:

I filed https://github.com/pypa/pip/issues/9601 for this since it seems like it's well outside of our control. If anyone else can add more information that would be great.

Thanks for doing the analysis and filing that bug. I didn't want to jump on anything, and wanted to give some breathing room for folks to look at that and chime in. Wilfredo has this PR to klein where he is also looking at these issues: https://github.com/twisted/klein/pull/450 So I think that I will not hold up the Twisted core release based on this issue. Hopefully this issue can be resolved outside of Twisted core. -- Craig

Reply

Sign in to reply online Use email software

Adi Roiban

12 Feb 12 Feb

2:45 a.m.

On Thu, 11 Feb 2021 at 19:26, Wilfredo Sánchez Vega wrote:

On Feb 10, 2021, at 3:44 PM, Glyph wrote:

This isn't even a bug in Klein, it's an issue with a version pin in its tox.ini: https://github.com/twisted/klein/blob/6e7b37158dea2fe73180809803a872ed98143c...

The constraints from requests (<3,>=2.5) and hyperlink (>=2.5) are perfectly compatible; one's just a subset of the other.

Sure, they are compatible, but Klein doesn’t use requests directly, and this only looks to be failing in trunk.

My concern here is that Twisted may have added a dependency on requests, and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini does, that you aren’t allowed to. That seems weak; I’d say a bug. No?

I have created a new virtual env and executed `pip install .` on trunk. It installed the following packages: constantly, zope.interface, attrs, incremental, six, Automat, idna, hyperlink, Twisted ------------ `requests` was not installed. I wasn't expecting to see `idna` as it is listed as TLS only deps.... but it looks like hyperlink depends on idna. Here is the dep tree $ pipdeptree -fl Twisted @ file:///home/adi/dev/twisted attrs==20.3.0 Automat==20.2.0 attrs==20.3.0 six==1.15.0 constantly==15.1.0 hyperlink==21.0.0 idna==3.1 incremental==17.5.0 zope.interface==5.2.0 setuptools==44.0.0 wheel==0.36.2 ------------ I guess that we can remove idna from setup.cfg TLS section in Twisted tls = pyopenssl >= 16.0.0 # service_identity 18.1.0 added support for validating IP addresses in # certificate subjectAltNames service_identity >= 18.1.0 idna >= 2.4 Cheers -- Adi Roiban

Reply

Sign in to reply online Use email software

Glyph

3:36 a.m.

On Feb 11, 2021, at 1:15 PM, Adi Roiban wrote:

I guess that we can remove idna from setup.cfg TLS section in Twisted

Nope; we use it directly, in https://github.com/twisted/twisted/blob/7cf6c8bc320ac5fd96b4784f6feb932ea819... https://github.com/twisted/twisted/blob/7cf6c8bc320ac5fd96b4784f6feb932ea819.... In fact, if anything, we should bump it up, since we use it even in non-tls configurations: https://github.com/twisted/twisted/blob/cd97222df2ca7032bbff2fe9a8793d7b42de... https://github.com/twisted/twisted/blob/cd97222df2ca7032bbff2fe9a8793d7b42de... . -g

Reply

Sign in to reply online Use email software

1166

Age (days ago)

1169

Last active (days ago)

Download

9 comments

4 participants

tags

participants (4)

Adi Roiban
Craig Rodrigues
Glyph
Wilfredo Sánchez Vega