[Distutils] Fwd: What is the official position on distutils?

[Nick Coghlan]

On 5 September 2016 at 16:47, Sylvain Corlay <sylvain.corlay at gmail.com> wrote:
> Hi Nick (re-sending to the list)

(Sylvain's seen some of what I say below in off-list replies, but I
figured it made sense to go over my point of view again for the list's
benefit)

> On Sun, Sep 4, 2016 at 5:16 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>>
>> It's important to note that one of our key goals in PyPA is to get to
>> a point where the API of a developer's publishing toolchain of choice
>> only depends on the *version of the publishing tools* they're using,
>> and not on the specific version of Python (as long as they're using a
>> supported version). Encouraging the expansion and increased use of the
>> distutils API runs counter to that goal in most cases (since it
>> encourages writing publication and distribution code that will only
>> work with newer versions of Python), but may sometimes make sense if
>> it's info that really is CPython centric, or if it's a feature
>> primarily aimed at tightly controlled environments that can
>> aggressively drop support for older versions.
>
>
> Today's state of things is that any project that is not a trivial
> pure-python module often has use of a combination of all three libraries:
> distutils, setuptools and pip in their setup.py and in  a convoluted way.
> Newcomers to the toolchain look in similar projects for the right
> incantations that will fulfill their needs. We are pretty far from being in
> a state where good practices can even be defined.

One of the good practices that *is* defined is "Use setuptools rather
than distutils" (footnote [5] gives the rationale):
https://packaging.python.org/current/#packaging-tool-recommendations

That's one of the key reasons we want to ensure setuptools
consistently provides a superset of distutils functionality, and also
ensure that any new functionality is available on older versions of
Python, either by monkeypatching distutils from setuptools (when
absolutely necessary due to the way distutils works), or by providing
version independent equivalent APIs in setuptools itself (when it's
sufficient to give publishers access to the functionality from
setup.py, rather than implicitly altering the behaviour of distutils
itself).

> As a package author, I generally make sure that my need is not fulfilled by
> a proper usage of the standard library before adding a dependency to a
> third-party library, but packaging is an area where this does not work.

This is now by design, as PyPA's view is that coupling the lifecycle
of a language ecosystem's publishing toolchain to the reference
implementation of the language runtime is a bad idea, since publishing
tools need to be free to update in response to changes in supported
target environments, without necessarily dropping support for older
versions of the language runtime itself.

PEP 518 (declarative dependencies for setup.py), and the competing
PEPs 516 & 517 for pluggable build system invocation are designed to
better enable that, although Daniel Holth's work with enscons shows
that it's possible to bootstrap an existing external build system from
setup.py, even without formal support for that at the interoperability
specification level. The reason packaging in particular is insistent
on the use of documented interoperability standards is that part of
the goal is to drop the requirement that build systems for Python
projects must be written in Python - they may need or benefit from
offering Python bindings, but working with our data formats in other
languages should be fully supported.

> Requiring pip only to know where distutils puts assets is pretty bad.

This is an area that "just" requires funding and contributions -
distlib (pragmatic, not necessarily standards based) and packaging
(strictly based on agreed interoperability standards) both exist to
serve as pattern extraction libraries for pip, PyPI, setuptools,
distutils, sysconfig etc, and make the things they do internally more
readily available to other software.

However, pursuing this model effectively requires giving up on the
idea that distutils will ever be "fixed" - it won't. There are deep
structural flaws in the design of its API (which even Greg Ward, its
original creator, now considers a bad idea) that can't be addressed
without major compatibility breaks, so it makes more sense to enable
and pursue the "strangler application" model [1], where we facilitate
the adoption of alternative build systems by publishers in a way
that's transparent to software consumers, and then base future
recommendations for default tooling choices based on emergent
community behaviour. The focus for distutils becomes "don't break
currently working build processes", while setuptools and other PyPA
projects take up the challenge of driving improvements in the
capabilities made available to publishers.

At the end of that process (which is likely to take years), there
shouldn't be any software build activities left using plain distutils
- everything that hasn't switched to a different build system entirely
will have either switched explicitly to setuptools, or will have
setuptools injected into the build process by build environments the
same way pip already does.

[1] http://www.martinfowler.com/bliki/StranglerApplication.html

> Interestingly, on the subject of building extensions (where the has_flag
> method discussion applies), distutils.ccompiler is the only available tool
> and happens to not be monkey-patched by setuptools at the moment. Although
> setuptools re-defines buildext (and tries to get a version from cython
> first!) So it seems that there no real "right place" within setuptools to
> monkey-patch distutils' ccompiler.

If I understand the change correctly, the new feature would only be
used explicitly, and thus may not require monkeypatching at all.
Instead, it could just require importing buildext from setuptools
rather than distutils if someone wants the new behaviour on older
Python versions.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia