[Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc

[Robert Collins]

I'm a little over this particular subthread of the topic - we did it
to death late last year. So apologies in advance if I get a little
terse.

On 12 February 2016 at 05:08, M.-A. Lemburg <mal at egenix.com> wrote:
> On 10.02.2016 19:46, Robert Collins wrote:
>> On 11 February 2016 at 02:36, M.-A. Lemburg <mal at egenix.com> wrote:
>>
>>>> Currently what pip does is to
>>>> invoke
>>>>
>>>>     $ python setup.py egg_info --egg-base $TEMPDIR
>>>>
>>>> to get the metadata. It is not possible to get the metadata without
>>>> executing the setup.py which is problematic for many applications.
>>>> Providing a static pypa.json file is much better: tools can read a
>>>> static file to get the metadata.
>>>
>>> Depends on which kind of meta data you're after. sdist packages
>>> do include the static PKG-INFO file which has the version 1.0
>>> meta data. This doesn't include dependencies or namespace
>>> details, but it does have important data such as version,
>>> package name, description, etc.
>>
>> For pip to use it, it needs to include - reliably - version, name, and
>> dependencies; for it to be in any way better, we also need
>> setup-requires or a functional equivalent.
>>
>> Today, we can't rely on the PKG-INFO being complete, so we assume they
>> are all wrong and start over. One of the things we'll get by being
>> strict in subsequent iterations is the ability to rely on things.
>
> Then why not fix distutils' sdist command to add the needed
> information to PKG-INFO and rely on it ?

How can we tell it is reliable? There's some 100K's of sdists that
aren't reliable on PyPI already. We don't want to break installing
those.

> Or perhaps add a new distutils command which outputs the needed
> information as JSON file and fix the sdist command to call this
> by default ?

We already have a command which outputs the needed info (as egg info)
- and my draft PEP has a similar one, using PEP427 wheel METADATA
format.

> There are many ways to address such issues, but defining a new
> standard for every issue we have instead of fixing the existing
> distutils implementation is not the best way to approach this.

I think you need to make a much stronger case for this, given how
consistent the disagreement in this forum with that position has been.
I understand you consider it to be not-the-best way, but so far, noone
else seems to agree.

Specifics that Donald raised with any 'fix distutils' approach:
 - how do we fix people running pip install on Python 2.7, 3.4, 3.5,
for the next 5 years?
 - how do we address the needs of folk who are using bento or flit
 - and how do we address the needs of the *authors* of bento or flit?

Plus the one I've already mentioned

 - how do we fix the bootstrap issue setup.py has ?

>>> In the end, you'll just be defining a different standard
>>> to express the same thing in different ways.
>>>
>>> The setup.py interface was never designed with integration in mind
>>> (only with the idea to provide an extensible interface; and I'm not
>>> going get into the merits of setuptools additions such as
>>> --single-version-externally-managed :-)) but it's still
>>> quite usable for the intended purpose.
>>
>> However we *are defining an integration point*, which is yet another
>> reason not to use the setup.py interface.
>
> https://xkcd.com/927/ :-)
>
> setup.py is the current standard and even though it's not
> necessarily nice, it works well and it does support adding
> different build systems (among many other things).

I don't think 'well' is a useful adjective here. setup.py has some
very sharp limits as an interface:
 - bootstrap dependencies are broken in non-standard networking environments
 - bootstrap dependencies are very slow due to multiple builds
 - setup.py is very complex for both implementors and package authors

Folk that want/need/like setup.py can keep using it! The draft we're
discussing can obviously thunk back through to setup.py for setuptools
projects, and no harm occurs to them. For other projects, they gain
choice and the ecosystem can expand.

> mxSetup.py, for example, includes a build system for C libraries
> completely outside the distutils system, based on the standard
> Unix configure/make dance. It simply hooks into distutils, takes
> a few parameters and goes off to build things, feeding the
> results back into the distutils machinery.

Thats great; and mxSetup.py can keep doing that, while adding compat
for pypa.json very easily.

Can I ask - why the push-back on a new, clean, crisp, separate
interface here? All the drafts had this in common, perhaps we're just
too close to it!

What harm is it going to cause?

-Rob



-- 
Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud