Pipenv’s resolver does not use troove classifiers as far as I am aware. I could be missing something though, the current resolver implementation is not the best code base to reason with. Can you provide some ideas on this, Dan? TP

On 20/9, 2018, at 15:09, Bernat Gabor wrote:

One of such differences is pipenv apparently trying to work from the troove classifiers for version compatibility instead of the existing python_requires (see https://github.com/tox-dev/tox/pull/1005 https://github.com/tox-dev/tox/pull/1005). Why? Point in case how people start to think pipenv is pip. At this point, pip is a completely different beast than pip I think.

Such check at best would fall under the ``twine check`` command at PyPi upload time, not at install time.

On Thu, Sep 20, 2018 at 8:03 AM Tzu-ping Chung mailto:uranusjr@gmail.com> wrote:

...

On 20/9, 2018, at 13:22, Chris Jerdonek mailto:chris.jerdonek@gmail.com> wrote:

On Wed, Sep 19, 2018 at 8:54 PM, Dan Ryan mailto:dan@danryan.co> wrote: I should clarify that we have already implemented a number of these as libraries over the last several months (and I am super familiar with pip's internals by now and I'm sure TP is getting there as well). More on this below ... We are super cognizant of that aspect as I am pretty sure we are hitting this wall in a full (nearly) pip-free reimplementation of all of the pipenv internals from the ground up, including wheel building/installation, but we basically had to start by calling pip directly, then slowly reimplement each aspect of the underlying logic using various elements in distlib/setuptools or rebuilding those.

Is the hope or game plan then for pipenv not to have to depend on pip? This is partly what I was trying to learn in my email to this list a month ago (on Aug. 20, with subject: "pipenv and pip"): https://mail.python.org/mm3/archives/list/distutils-sig@python.org/thread/2Q... https://mail.python.org/mm3/archives/list/distutils-sig@python.org/thread/2Q...

Based on the replies, I wasn't getting that impression at the time (though I don't remember getting a clear answer), but maybe things have changed since then.

The resolution side of Pipenv really needs a Python API, and also cannot really use the CLI because it needs something slightly different than pip’s high-level logic (Nick mentioned this briefly). If we can’t use pip internals, then yes, the plan is to not depend on pip. The hope is we can share those internals with pip (either following the same standards, or using the same implementation), hence my series of questions.

The installation side of Pipenv will continue to use pip directly, at least for a while more even after the resolution side breaks away, since “pip install” is adequate enough for our purposes. There are some possible improvements if there is a lower-layer library (e.g. to avoid pip startup overhead), but that is far less important.

...

It should certainly be a lot easier for pipenv to move fast since there is no legacy base of users to maintain compatibility with. However, I worry about the fracturing this will cause. In creating these libraries, from the pip tracker it doesn't look like any effort is going into refactoring pip to make use of them. This relates to the point I made earlier today about how there won't be an easy way to cut pip over to using a new library unless an effort is made from the beginning. Thus, it's looking like things could be on track to split the user and maintainer base in two, with pip bearing the legacy burden and perhaps not seeing the improvements. Are we okay with that future?

I’m afraid the new implementation will still need to deal with compatibility issues. Users expect Pipenv to work exactly as pip, and get very angry if it does not, especially when they see it is under the PyPA organisation on GitHub. The last time Pipenv tries to explain it does whatever arbitrary things it does, we get labelled as “toxic” (there are other issues in play, but this is IMO the ultimate cause). Whether the image is fair or not, I would most definitely want to avoid similar incidents from happening again.

I think Pipenv would be okay to maintain a different (from scratch) implementation than pip’s, but it would still need to do (almost) exactly what pip is doing, unless we can have people (pip maintainers or otherwise) backing the differences. Whether pip uses the new implementation or not wouldn’t change the requirement :(

TP

...

--Chris

Since you mentioned following along, here's what we're working on right now:

https://github.com/sarugaku/requirementslib https://github.com/sarugaku/requirementslib -- abstraction layer for parsing and converting various requirements formats (pipfile/requirements.txt/command line/InstallRequirement) and moving between all of them https://github.com/sarugaku/resolvelib https://github.com/sarugaku/resolvelib -- directed acyclic graph library for handling dependency resolution (not yet being used in pipenv) https://github.com/sarugaku/passa https://github.com/sarugaku/passa -- dependency resolver/installer/pipfile manager (bulk of the logic we have been talking about is in here right now) -- I think we will probably split this back out into multiple other smaller libraries or something based on the discussion https://github.com/sarugaku/plette https://github.com/sarugaku/plette -- this is a rewrite of pipfile with some additional logic / validation https://github.com/sarugaku/shellingham https://github.com/sarugaku/shellingham -- this is a shell detection library made up of some tooling we built in pipenv for environment detection https://github.com/sarugaku/pythonfinder https://github.com/sarugaku/pythonfinder -- this is a library for finding python (pep 514 compliant) by version and for finding any other executables (cross platform) https://github.com/sarugaku/virtenv https://github.com/sarugaku/virtenv -- python api for virtualenv creation

Happy to provide access or take advice as needed on any of those. Thanks all for the receptiveness and collaboration

Dan Ryan gh: @techalchemy // e: dan@danryan.co mailto:dan@danryan.co

From: Donald Stufft [mailto:donald@stufft.io mailto:donald@stufft.io] Sent: Wednesday, September 19, 2018 1:52 PM To: Tzu-ping Chung Cc: Distutils Subject: [Distutils] Re: Distlib vs Packaging (Was: disable building wheel for a package)

My general recommendation if you want a Python implementation/interface for something pip does, is:

- Open an issue on the pip repository to document your intent and to make sure that there is nobody there who is against having that functionality split out. This might also give a chance for people with familiarity in that API to mention pain points that you can solve in a new API. We can also probably give you a good sense if the thing you want in a library is something that probably has multiple things that are dependent on getting split out first (for instance, if you said you wanted a library for installing wheels, we'd probably tell you that there is a dependency on PEP 425 tags, pip locations, maybe other that need resolved first) and also whether this is something that should have a PEP first or not. Getting some rough agreement on the plan to split X thing out before you start is overall a good thing.

- Create or update a PEP if required, and get it into the provisional state.

- Make the library, either as a PR to packaging or as it's own independent library. If there are questions that come up while creating that library/PR that have to do with specific pip behaviors, go back to that original issue and ask for clarification etc. Ideally at some point you'll open a PR on pip that uses the new library (my suggestion is to not bundle the library in the initial PR, and just import it normally so that the PR diff doesn't include the full bundled library until there's agreement on it). If there's another tool (pipenv, whatever) that is looking to use that same functionality, open a WIP PR there too that switches it to using that. Use feedback and what you learn from trying to integrate in those libraries to influence back the design of the API itself.

Creating a PEP and creating the library and the PRs can happen in parallel, but at least for pip if something deserves a PEP, we're not going to merge a PR until that PEP is generally agreed on. However it can be supremely useful to have them all going at the same time, because you run into things that you didn't really notice until you went to actually implement it.

My other big suggestion would be to e careful about how much you bite off at one time. Pip's internal code base is not the greatest, so pulling out smaller chunks at a time rather than trying to start right off pulling out a big topic is more likely to meet with success. Be cognizant of what the dependencies are for the feature you want to implement, because if it has dependencies, you'll need to pull them out first before you can pull it out OR you'll need to design the API to invert those dependencies so they get passed in instead.

I personally would be happy to at a minimum participate on any issue where someone was trying to split out some functionality from pip into a re-usable library if not follow the develop of that library directly to help guide it more closely. My hope for pip is that it ends up being the glue around a bunch of these libraries, and that it doesn't implement most of the stuff itself anymore. -- Distutils-SIG mailing list -- distutils-sig@python.org mailto:distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org mailto:distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/I... https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/I...

-- Distutils-SIG mailing list -- distutils-sig@python.org mailto:distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org mailto:distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/C... https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/C...