[Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc

Robert Collins robertc at robertcollins.net
Wed Feb 10 13:42:02 EST 2016

On 11 February 2016 at 01:21, M.-A. Lemburg <mal at egenix.com> wrote:
> On 10.02.2016 12:10, Paul Moore wrote:
>> On 10 February 2016 at 10:23, M.-A. Lemburg <mal at egenix.com> wrote:
>>> IMO, that's easy to achieve, though, with the existing de-facto
>>> standard interface we already have: the setup.py command line API.
>>> We'd just need to publish the minimal set of commands and options,
>>> installer will want to see implemented in order to initiate
>>> the builds.
>> No-one who's investing time in writing PEPs is willing to thrash out
>> the details of how to use the setup.py interface in a formal proposal
>> that sticks to the sort of "minimum required" spec that alternative
>> tools would be willing to support. And there's no indication that tool
>> developers are willing to implement a setup.py compatible interface
>> format as you suggest. And finally, you'd need a way to declare that
>> pip installs tool X before trying to run setup.py.
> I don't think that installing 3rd party tools is within the scope
> of such a proposal. The setup.py of packages using such tools would
> have to either define a dependency to have the installer get the
> extra tool, download and install it directly when needed, or tell
> the user how to install the tool.
> Alternatively, the package distro could simply ship the tool
> embedded in the package. That's what we're doing with
> mxSetup.py.
>> So "easy to achieve" still needs someone to take the time to deal with
>> these sorts of issue. It's the usual process of the people willing to
>> put in the effort get to choose the direction (which is also why I
>> just provide feedback, and don't tend to offer my own proposals,
>> because I'm not able to commit that sort of time).
> Wait. You are missing the point that the setup.py interface
> already does work, so no extra effort is needed. All that's
> needed is some documentation of what's currently being used,
> so that other tools can support the interface going forward.
> At the moment, pip this interface is only defined by
> "what pip uses" and that's a moving target.

I disagree with the claim that setup.py already works.

Right now the vast majority of setup.py's will end up invoking
easy-install, which has entirely separate configuration to pip for
networking. It also means that pip is utterly incapable of caching and
reusing setup_requires dependencies, so when e.g. numpy is a
setup-requires dependency, build times can be arbitrarily high as
numpy gets built 3, 4, 5 or more times.

Secondly, the setup.py interface includes 'install' as a verb, and
there is pretty solid consensus that that is a bug - any definition of
'what pip uses' has to include that as a fallback, for the same
reasons pip has to fallback to that - but there is a substantial
difference between the end user UX setuptools provides, the interface
pip is forced to use, and the smaller one pip would /like/ to use.

So all the variants of the PEP we've been discussing are about a
modest step forward, capturing pip's existing needs and addressing the
setup-requires/easy-install bug as well as the use of install bug.


Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud

More information about the Distutils-SIG mailing list