The alpha or beta release of pip with its new dependency resolver should be out in May. I just posted https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.... which discusses what is going to change in the pip resolver, when, and how you can help (including some low-effort things you can do right now). I didn't mention this in the blog post because ordinary Python users shouldn't try it, but: As of right now, people who install pip from GitHub master will have the ability to run `pip install --unstable-feature=resolver` and test the new resolver code. And less than half of the test suite fails! Expect errors and missing features, but it’s there! [Celebratory trumpet honk here.] Hope all of you, and all the people you are close to, are healthy and staying that way. -- Sumana Harihareswara pip project manager (contracting with Python Software Foundation) Changeset Consulting https://changeset.nyc
I couldn't find this in the blog post but is the plan to make the resolver a separate package so other tools can use it? Or is the plan perhaps to get it working in pip first and to break it out later?
It's already available as a separate package: https://pypi.org/project/resolvelib/ Paul On Tue, 24 Mar 2020 at 18:52, Brett Cannon <brett@python.org> wrote:
I couldn't find this in the blog post but is the plan to make the resolver a separate package so other tools can use it? Or is the plan perhaps to get it working in pip first and to break it out later? -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/CLWGX...
IIRC, the issue for this (or one of the issues for this great work) is: "New Resolver: Rollout, Feedback Loops and Development Flow" https://github.com/pypa/pip/issues/6536 On Tue, Mar 24, 2020 at 3:03 PM Paul Moore <p.f.moore@gmail.com> wrote:
It's already available as a separate package: https://pypi.org/project/resolvelib/
Paul
On Tue, 24 Mar 2020 at 18:52, Brett Cannon <brett@python.org> wrote:
I couldn't find this in the blog post but is the plan to make the
resolver a separate package so other tools can use it? Or is the plan perhaps to get it working in pip first and to break it out later?
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/CLWGX... -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/43NVR...
I think it may be most likely that you'll get quality feedback through GitHub Issues. Is there an issue template with at least some of the 23 questions in the survey? Survey: https://tools.simplysecure.org/survey/index.php?r=survey/index&sid=989272&lang=en Here's the "Resolver refactorings" project board: https://github.com/pypa/pip/projects/5 A note in the README and the docs regarding testing the new resolver might be good too. On Tue, Mar 24, 2020 at 3:55 PM Wes Turner <wes.turner@gmail.com> wrote:
IIRC, the issue for this (or one of the issues for this great work) is: "New Resolver: Rollout, Feedback Loops and Development Flow" https://github.com/pypa/pip/issues/6536
On Tue, Mar 24, 2020 at 3:03 PM Paul Moore <p.f.moore@gmail.com> wrote:
It's already available as a separate package: https://pypi.org/project/resolvelib/
Paul
On Tue, 24 Mar 2020 at 18:52, Brett Cannon <brett@python.org> wrote:
I couldn't find this in the blog post but is the plan to make the
resolver a separate package so other tools can use it? Or is the plan perhaps to get it working in pip first and to break it out later?
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/CLWGX... -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/43NVR...
Hi, Wes! Thanks for your note! We'll look into adjusting our issue templates. But -- not sure whether you've read my piece https://www.harihareswara.net/sumana/2017/04/07/1 "Inclusive-Or: Hospitality in Bug Tracking", but it's really important to expand how we get feedback from our users beyond people who are already comfortable using GitHub. And our user experience experts at Simply Secure are using the survey to find out not just what issues people currently have, but also who's available for more in-depth discussions as we progress. So we're going to pay attention both to the survey responses and to issues that come in via GitHub. Sounds like you'll be submitting a pull request for that documentation & README change -- go ahead and ping me as @brainwane for review when you do that! Thanks. -- Sumana Harihareswara Changeset Consulting https://changeset.nyc On 3/24/20 4:07 PM, Wes Turner wrote:
I think it may be most likely that you'll get quality feedback through GitHub Issues. Is there an issue template with at least some of the 23 questions in the survey?
Survey: https://tools.simplysecure.org/survey/index.php?r=survey/index&sid=989272&lang=en
Here's the "Resolver refactorings" project board: https://github.com/pypa/pip/projects/5
A note in the README and the docs regarding testing the new resolver might be good too.
On Tue, Mar 24, 2020 at 3:55 PM Wes Turner <wes.turner@gmail.com> wrote:
IIRC, the issue for this (or one of the issues for this great work) is: "New Resolver: Rollout, Feedback Loops and Development Flow" https://github.com/pypa/pip/issues/6536
On Tue, Mar 24, 2020 at 3:03 PM Paul Moore <p.f.moore@gmail.com> wrote:
It's already available as a separate package: https://pypi.org/project/resolvelib/
Paul
On Tue, 24 Mar 2020 at 18:52, Brett Cannon <brett@python.org> wrote:
I couldn't find this in the blog post but is the plan to make the
resolver a separate package so other tools can use it? Or is the plan perhaps to get it working in pip first and to break it out later?
On 3/24/20 4:13 PM, Sumana Harihareswara wrote:
Sounds like you'll be submitting a pull request for that documentation & README change -- go ahead and ping me as @brainwane for review when you do that! Thanks.
Sorry, that last bit was kinda snarky. Am making this change myself now. -- Sumana Harihareswara Changeset Consulting https://changeset.nyc
Issue and Pull Request Templates: - https://github.com/devspace/awesome-github-templates - Sometimes, a blank issue template option seems most welcoming When users have issues with the resolver, the most helpful thing for them to do may be to add a test case. IDK what the best way to do that would be? - https://github.com/sarugaku/resolvelib/tree/master/tests/functional - Create a template test case directory? Or e.g. a YAML file to call pytest.mark.parametrize with? Ways to store comments in GitHub issues: - https://github.com/utterance/utterances - https://github.com/imsun/gitment On Tue, Mar 24, 2020 at 4:21 PM Sumana Harihareswara <sh@changeset.nyc> wrote:
On 3/24/20 4:13 PM, Sumana Harihareswara wrote:
Sounds like you'll be submitting a pull request for that documentation & README change -- go ahead and ping me as @brainwane for review when you do that! Thanks.
Sorry, that last bit was kinda snarky. Am making this change myself now. -- Sumana Harihareswara Changeset Consulting https://changeset.nyc
A few more project management resources (A bit OT, but intended to be helpful:) ## Ways to integrate GitHub with other data sources: ### GitHub APIs: - https://developer.github.com/v3/ - https://developer.github.com/v4/ - https://developer.github.com/v3/libraries/ - https://github.com/cli/cli - https://github.com/github/hub - https://hub.github.com/#developer ### IFTTT - https://ifttt.com/github - Create a Github issue when there's a new vulnerability published ### Zapier - https://zapier.com/apps/github/integrations - Create a GitHub issue when a new survey is posted <-- ### Bots - https://github.com/python/miss-islington - https://probot.github.io/apps/ - https://probot.github.io/apps/polls/ - https://probot.github.io/apps/triage-new-issues/ > Adds `triage` label to newly-created issues and then removes it when other label(s) are added ### Zulip - https://zulipchat.com/integrations/version-control - https://zulipchat.com/integrations/doc/github - https://zulipchat.com/integrations/doc/github_detail On Tue, Mar 24, 2020 at 5:18 PM Wes Turner <wes.turner@gmail.com> wrote:
Issue and Pull Request Templates: - https://github.com/devspace/awesome-github-templates - Sometimes, a blank issue template option seems most welcoming
When users have issues with the resolver, the most helpful thing for them to do may be to add a test case. IDK what the best way to do that would be? - https://github.com/sarugaku/resolvelib/tree/master/tests/functional - Create a template test case directory? Or e.g. a YAML file to call pytest.mark.parametrize with?
Ways to store comments in GitHub issues: - https://github.com/utterance/utterances - https://github.com/imsun/gitment
On Tue, Mar 24, 2020 at 4:21 PM Sumana Harihareswara <sh@changeset.nyc> wrote:
On 3/24/20 4:13 PM, Sumana Harihareswara wrote:
Sounds like you'll be submitting a pull request for that documentation & README change -- go ahead and ping me as @brainwane for review when you do that! Thanks.
Sorry, that last bit was kinda snarky. Am making this change myself now. -- Sumana Harihareswara Changeset Consulting https://changeset.nyc
To expand a little on the topic, there are multiple abstraction layers required to make the new resolver useful as a separate package. ResolveLib (mentioned in Paul’s message) is an dependency resolver implementation in abstract, entirely divorced of Python packaging. While it can be used by other tools, the implementer would need to provide the necessary parts to “teach” it how understand Python packages. The ongoing pip resolver work is exactly that, to integrate ResolveLib into pip by wiring it onto pip’s internal representation of Python packages. This is unfortunately not really reusable for other tools, since pip’s current model on Python packages is too deeply entangled into various parts of the code base. Some extra work is needed to make ResolveLib really useful as a reusable Python dependency resolver for other projects. One way would be to implement a wrapper library to provide the same Python package semantics to ResolveLib, but with more reusable models such as pypa/packaging, importlib-metadata, etc. The current pip resolver work would be able to offer much knowledge on dealing with various hairy problems, but the code written for pip wouldn’t be directly reusable. TP
On 25/3/2020, at 03:02, Paul Moore <p.f.moore@gmail.com> wrote:
It's already available as a separate package: https://pypi.org/project/resolvelib/
Paul
On Tue, 24 Mar 2020 at 18:52, Brett Cannon <brett@python.org> wrote:
I couldn't find this in the blog post but is the plan to make the resolver a separate package so other tools can use it? Or is the plan perhaps to get it working in pip first and to break it out later? -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/CLWGX...
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/43NVR...
On Tue, Mar 24, 2020 at 5:24 PM Tzu-ping Chung <uranusjr@gmail.com> wrote:
To expand a little on the topic, there are multiple abstraction layers required to make the new resolver useful as a separate package.
ResolveLib (mentioned in Paul’s message) is an dependency resolver implementation in abstract, entirely divorced of Python packaging. While it can be used by other tools, the implementer would need to provide the necessary parts to “teach” it how understand Python packages.
The ongoing pip resolver work is exactly that, to integrate ResolveLib into pip by wiring it onto pip’s internal representation of Python packages. This is unfortunately not really reusable for other tools, since pip’s current model on Python packages is too deeply entangled into various parts of the code base.
Some extra work is needed to make ResolveLib really useful as a reusable Python dependency resolver for other projects. One way would be to implement a wrapper library to provide the same Python package semantics to ResolveLib, but with more reusable models such as pypa/packaging, importlib-metadata, etc. The current pip resolver work would be able to offer much knowledge on dealing with various hairy problems, but the code written for pip wouldn’t be directly reusable.
TP
"New Resolver: technical choices" compares various resolvers. https://github.com/pypa/pip/issues/7406 FWICS, more test cases should be the immediate objective. Factoring out to make a generic resolver would be redundant? Would existing interfaces for e.g. libsolv (dnf) and pycosat (conda) be sufficient?
On 25/3/2020, at 03:02, Paul Moore <p.f.moore@gmail.com> wrote:
It's already available as a separate package: https://pypi.org/project/resolvelib/
Paul
On Tue, 24 Mar 2020 at 18:52, Brett Cannon <brett@python.org> wrote:
I couldn't find this in the blog post but is the plan to make the
resolver a separate package so other tools can use it? Or is the plan perhaps to get it working in pip first and to break it out later?
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/CLWGX... -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/43NVR... -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/VLSD5...
On 25/3/2020, at 05:37, Wes Turner <wes.turner@gmail.com> wrote:
On Tue, Mar 24, 2020 at 5:24 PM Tzu-ping Chung <uranusjr@gmail.com <mailto:uranusjr@gmail.com>> wrote: To expand a little on the topic, there are multiple abstraction layers required to make the new resolver useful as a separate package.
ResolveLib (mentioned in Paul’s message) is an dependency resolver implementation in abstract, entirely divorced of Python packaging. While it can be used by other tools, the implementer would need to provide the necessary parts to “teach” it how understand Python packages.
The ongoing pip resolver work is exactly that, to integrate ResolveLib into pip by wiring it onto pip’s internal representation of Python packages. This is unfortunately not really reusable for other tools, since pip’s current model on Python packages is too deeply entangled into various parts of the code base.
Some extra work is needed to make ResolveLib really useful as a reusable Python dependency resolver for other projects. One way would be to implement a wrapper library to provide the same Python package semantics to ResolveLib, but with more reusable models such as pypa/packaging, importlib-metadata, etc. The current pip resolver work would be able to offer much knowledge on dealing with various hairy problems, but the code written for pip wouldn’t be directly reusable.
TP
"New Resolver: technical choices" compares various resolvers. https://github.com/pypa/pip/issues/7406 <https://github.com/pypa/pip/issues/7406>
FWICS, more test cases should be the immediate objective.
Factoring out to make a generic resolver would be redundant? Would existing interfaces for e.g. libsolv (dnf) and pycosat (conda) be sufficient?
In theory yes, but the practical problem is that a few ideas in Python packaging can’t be matched cleanly to the general models. Extras (or conditional dependency in general) is one problem a lot of package managers have to deal with outside of the generic resolution process. Another problem is to identify where incompatibilities are declared—version specifiers would be the first coming to mind, but there are actually more, e.g. Requires-Python. All these are solvable problems, and indeed each problem can wire directly to the resolver interface. But they would then all need to solve with the same problems, and likely to each miss something in the process. I believe it would be immensely beneficial to have a “canonical” implementation all parties use and maintain, if only to keep the domain knowledge shared and kept.
On 25/3/2020, at 03:02, Paul Moore <p.f.moore@gmail.com <mailto:p.f.moore@gmail.com>> wrote:
It's already available as a separate package: https://pypi.org/project/resolvelib/ <https://pypi.org/project/resolvelib/>
Paul
On Tue, 24 Mar 2020 at 18:52, Brett Cannon <brett@python.org <mailto:brett@python.org>> wrote:
I couldn't find this in the blog post but is the plan to make the resolver a separate package so other tools can use it? Or is the plan perhaps to get it working in pip first and to break it out later? -- Distutils-SIG mailing list -- distutils-sig@python.org <mailto:distutils-sig@python.org> To unsubscribe send an email to distutils-sig-leave@python.org <mailto:distutils-sig-leave@python.org> https://mail.python.org/mailman3/lists/distutils-sig.python.org/ <https://mail.python.org/mailman3/lists/distutils-sig.python.org/> Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/CLWGX... <https://mail.python.org/archives/list/distutils-sig@python.org/message/CLWGXEPNNMZ6YI5WWJ3ZKWWY5WVQOCAE/>
-- Distutils-SIG mailing list -- distutils-sig@python.org <mailto:distutils-sig@python.org> To unsubscribe send an email to distutils-sig-leave@python.org <mailto:distutils-sig-leave@python.org> https://mail.python.org/mailman3/lists/distutils-sig.python.org/ <https://mail.python.org/mailman3/lists/distutils-sig.python.org/> Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/43NVR... <https://mail.python.org/archives/list/distutils-sig@python.org/message/43NVR5W7YKVFBDY5FSMBWQPLKKG2V2EP/> -- Distutils-SIG mailing list -- distutils-sig@python.org <mailto:distutils-sig@python.org> To unsubscribe send an email to distutils-sig-leave@python.org <mailto:distutils-sig-leave@python.org> https://mail.python.org/mailman3/lists/distutils-sig.python.org/ <https://mail.python.org/mailman3/lists/distutils-sig.python.org/> Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/VLSD5... <https://mail.python.org/archives/list/distutils-sig@python.org/message/VLSD5OXWR6YLAFXQZWCZSCLXPNUUBSU4/>
participants (5)
-
Brett Cannon -
Paul Moore -
Sumana Harihareswara -
Tzu-ping Chung -
Wes Turner