I should clarify that we have already implemented a number of these as
libraries over the last several months (and I am super familiar with pip's
internals by now and I'm sure TP is getting there as well). More on this
below
...
We are super cognizant of that aspect as I am pretty sure we are hitting
this wall in a full (nearly) pip-free reimplementation of all of the pipenv
internals from the ground up, including wheel building/installation, but we
basically had to start by calling pip directly, then slowly reimplement each
aspect of the underlying logic using various elements in distlib/setuptools
or rebuilding those.

Since you mentioned following along, here's what we're working on right now:

https://github.com/sarugaku/requirementslib -- abstraction layer for parsing
and converting various requirements formats
(pipfile/requirements.txt/command line/InstallRequirement) and moving
between all of them
https://github.com/sarugaku/resolvelib -- directed acyclic graph library for
handling dependency resolution (not yet being used in pipenv)
https://github.com/sarugaku/passa -- dependency resolver/installer/pipfile
manager (bulk of the logic we have been talking about is in here right now)
-- I think we will probably split this back out into multiple other smaller
libraries or something based on the discussion
https://github.com/sarugaku/plette -- this is a rewrite of pipfile with some
additional logic / validation
https://github.com/sarugaku/shellingham -- this is a shell detection library
made up of some tooling we built in pipenv for environment detection
https://github.com/sarugaku/pythonfinder -- this is a library for finding
python (pep 514 compliant) by version and for finding any other executables
(cross platform)
https://github.com/sarugaku/virtenv -- python api for virtualenv creation

Happy to provide access or take advice as needed on any of those.  Thanks
all for the receptiveness and collaboration

Dan Ryan
gh: @techalchemy // e: dan@danryan.co

From: Donald Stufft [mailto:donald@stufft.io]
Sent: Wednesday, September 19, 2018 1:52 PM
To: Tzu-ping Chung
Cc: Distutils
Subject: [Distutils] Re: Distlib vs Packaging (Was: disable building wheel
for a package)

My general recommendation if you want a Python implementation/interface for
something pip does, is:

- Open an issue on the pip repository to document your intent and to make
sure that there is nobody there who is against having that functionality
split out. This might also give a chance for people with familiarity in that
API to mention pain points that you can solve in a new API. We can also
probably give you a good sense if the thing you want in a library is
something that probably has multiple things that are dependent on getting
split out first (for instance, if you said you wanted a library for
installing wheels, we'd probably tell you that there is a dependency on PEP
425 tags, pip locations, maybe other that need resolved first) and also
whether this is something that should have a PEP first or not. Getting some
rough agreement on the plan to split X thing out before you start is overall
a good thing.

- Create or update a PEP if required, and get it into the provisional state.

- Make the library, either as a PR to packaging or as it's own independent
library. If there are questions that come up while creating that library/PR
that have to do with specific pip behaviors, go back to that original issue
and ask for clarification etc. Ideally at some point you'll open a PR on pip
that uses the new library (my suggestion is to not bundle the library in the
initial PR, and just import it normally so that the PR diff doesn't include
the full bundled library until there's agreement on it). If there's another
tool (pipenv, whatever) that is looking to use that same functionality, open
a WIP PR there too that switches it to using that. Use feedback and what you
learn from trying to integrate in those libraries to influence back the
design of the API itself.

Creating a PEP and creating the library and the PRs can happen in parallel,
but at least for pip if something deserves a PEP, we're not going to merge a
PR until that PEP is generally agreed on. However it can be supremely useful
to have them all going at the same time, because you run into things that
you didn't really notice until you went to actually implement it.

My other big suggestion would be to e careful about how much you bite off at
one time. Pip's internal code base is not the greatest, so pulling out
smaller chunks at a time rather than trying to start right off pulling out a
big topic is more likely to meet with success. Be cognizant of what the
dependencies are for the feature you want to implement, because if it has
dependencies, you'll need to pull them out first before you can pull it out
OR you'll need to design the API to invert those dependencies so they get
passed in instead.

I personally would be happy to at a minimum participate on any issue where
someone was trying to split out some functionality from pip into a re-usable
library if not follow the develop of that library directly to help guide it
more closely. My hope for pip is that it ends up being the glue around a
bunch of these libraries, and that it doesn't implement most of the stuff
itself anymore.

--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-leave@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/IQVZVVWX2BLEP6D4WQMKNXZHBF2NZINU/