[Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc

[Donald Stufft]

> On Feb 11, 2016, at 11:08 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
> Then why not fix distutils' sdist command to add the needed
> information to PKG-INFO and rely on it ?
> 
> Or perhaps add a new distutils command which outputs the needed
> information as JSON file and fix the sdist command to call this
> by default ?
> 
> There are many ways to address such issues, but defining a new
> standard for every issue we have instead of fixing the existing
> distutils implementation is not the best way to approach this.


The very nature of distutils (later inherited by setuptools) is the problem to
be honest. The reason we're adding new standards and moving away from these
systems is that fixing them is essentially fundamentally altering them.

For instance, adding some new information to PKG-INFO or turning it into a json
file doesn't address the fundamental problem with why we can't trust the
metadata. The reason we can't trust the metadata is because setup.py means that
the metadata can (and does!) change based on what system you're executing the
setup.py on. Here's a common pattern:


    import sys

    from setuptools import setup

    install_requires = []

    if sys.version_info[:2] < (2,7):
        install_requires.append("argparse")

    setup(..., install_requires=install_requires, ...)


Any static metadata that is generated by that setup.py is going to change based
on what version of Python you're executing it under. This isn't something you
can just sort of replace, the setup.py *is* the "source of truth" for this
information and as long as it is, we can't trust a byproduct of executing that
file.

In addition, the idea that a singular build system is really the best fit for
every situation is I think, fairly nieve. Even today we have multiple build
systems (such as numpy.distutils) even though utilizing them is actually fairly
painful. This speaks to me that the need is fairly strong since people are
willing to put up with that pain in order to swap out distutils/setuptools for
something else.

As far as whether setup.py or something else should be the integration point
I don't think that either choice would be a bad one. However I prefer using
something else for a few reasons:

* The setup.py interface is completely entangled with the implementation of
  distutils and setuptools. We can't change anything about it because of it
  being baked into the Python standard library.

* A script that is executed as part of the packaging of the project is
  notoriously hard to test. The best layout if we make setup.py the integration
  point is that the setup.py will be a small shim that will call into some
  other bit of code to do it's work. At that point though, why bother with the
  shim instead of just calling that code directly?

* Having the script be part of the project encourages small, project specific
  one off hacks. These hacks have a tendency to be fragile and they regularly
  break down. Having the build tool be packaged externally and pulled in tends
  to encourage reusable code.

That's on top of the *other* problem, that we can't fundamentally change
distutils in anything but Python 3.6 at this point. That's not a terribly
great place to be in though, because packaging has network effects and nobody
is going to drop support for 2.7 (209,453,609 downloads from PyPI since Jan 14)
just to make packaging their library slightly nicer. Even 2.6 (154,27,345) has
more of the marketshare of what is downloading from PyPI than any version of
Python 3.x. Moving things out of the standard library is how we've been able to
progress packaging as quickly and as universally as we have been in the past
few years. I am very opposed to any plan of improvement that involves waiting
around on 3.6 to be the minimum version of Python.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160211/16a0354a/attachment.sig>