[Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc
donald at stufft.io
Thu Feb 11 11:48:18 EST 2016
> On Feb 11, 2016, at 11:08 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> Then why not fix distutils' sdist command to add the needed
> information to PKG-INFO and rely on it ?
> Or perhaps add a new distutils command which outputs the needed
> information as JSON file and fix the sdist command to call this
> by default ?
> There are many ways to address such issues, but defining a new
> standard for every issue we have instead of fixing the existing
> distutils implementation is not the best way to approach this.
The very nature of distutils (later inherited by setuptools) is the problem to
be honest. The reason we're adding new standards and moving away from these
systems is that fixing them is essentially fundamentally altering them.
For instance, adding some new information to PKG-INFO or turning it into a json
file doesn't address the fundamental problem with why we can't trust the
metadata. The reason we can't trust the metadata is because setup.py means that
the metadata can (and does!) change based on what system you're executing the
setup.py on. Here's a common pattern:
from setuptools import setup
install_requires = 
if sys.version_info[:2] < (2,7):
setup(..., install_requires=install_requires, ...)
Any static metadata that is generated by that setup.py is going to change based
on what version of Python you're executing it under. This isn't something you
can just sort of replace, the setup.py *is* the "source of truth" for this
information and as long as it is, we can't trust a byproduct of executing that
In addition, the idea that a singular build system is really the best fit for
every situation is I think, fairly nieve. Even today we have multiple build
systems (such as numpy.distutils) even though utilizing them is actually fairly
painful. This speaks to me that the need is fairly strong since people are
willing to put up with that pain in order to swap out distutils/setuptools for
As far as whether setup.py or something else should be the integration point
I don't think that either choice would be a bad one. However I prefer using
something else for a few reasons:
* The setup.py interface is completely entangled with the implementation of
distutils and setuptools. We can't change anything about it because of it
being baked into the Python standard library.
* A script that is executed as part of the packaging of the project is
notoriously hard to test. The best layout if we make setup.py the integration
point is that the setup.py will be a small shim that will call into some
other bit of code to do it's work. At that point though, why bother with the
shim instead of just calling that code directly?
* Having the script be part of the project encourages small, project specific
one off hacks. These hacks have a tendency to be fragile and they regularly
break down. Having the build tool be packaged externally and pulled in tends
to encourage reusable code.
That's on top of the *other* problem, that we can't fundamentally change
distutils in anything but Python 3.6 at this point. That's not a terribly
great place to be in though, because packaging has network effects and nobody
is going to drop support for 2.7 (209,453,609 downloads from PyPI since Jan 14)
just to make packaging their library slightly nicer. Even 2.6 (154,27,345) has
more of the marketshare of what is downloading from PyPI than any version of
Python 3.x. Moving things out of the standard library is how we've been able to
progress packaging as quickly and as universally as we have been in the past
few years. I am very opposed to any plan of improvement that involves waiting
around on 3.6 to be the minimum version of Python.
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
More information about the Distutils-SIG