On 5 September 2016 at 16:47, Sylvain Corlay <sylvain.corlay@gmail.com> wrote:

Hi Nick (re-sending to the list)

(Sylvain's seen some of what I say below in off-list replies, but I figured it made sense to go over my point of view again for the list's benefit)

On Sun, Sep 4, 2016 at 5:16 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

It's important to note that one of our key goals in PyPA is to get to a point where the API of a developer's publishing toolchain of choice only depends on the *version of the publishing tools* they're using, and not on the specific version of Python (as long as they're using a supported version). Encouraging the expansion and increased use of the distutils API runs counter to that goal in most cases (since it encourages writing publication and distribution code that will only work with newer versions of Python), but may sometimes make sense if it's info that really is CPython centric, or if it's a feature primarily aimed at tightly controlled environments that can aggressively drop support for older versions.

Today's state of things is that any project that is not a trivial pure-python module often has use of a combination of all three libraries: distutils, setuptools and pip in their setup.py and in a convoluted way. Newcomers to the toolchain look in similar projects for the right incantations that will fulfill their needs. We are pretty far from being in a state where good practices can even be defined.

One of the good practices that *is* defined is "Use setuptools rather than distutils" (footnote [5] gives the rationale): https://packaging.python.org/current/#packaging-tool-recommendations That's one of the key reasons we want to ensure setuptools consistently provides a superset of distutils functionality, and also ensure that any new functionality is available on older versions of Python, either by monkeypatching distutils from setuptools (when absolutely necessary due to the way distutils works), or by providing version independent equivalent APIs in setuptools itself (when it's sufficient to give publishers access to the functionality from setup.py, rather than implicitly altering the behaviour of distutils itself).

As a package author, I generally make sure that my need is not fulfilled by a proper usage of the standard library before adding a dependency to a third-party library, but packaging is an area where this does not work.

This is now by design, as PyPA's view is that coupling the lifecycle of a language ecosystem's publishing toolchain to the reference implementation of the language runtime is a bad idea, since publishing tools need to be free to update in response to changes in supported target environments, without necessarily dropping support for older versions of the language runtime itself. PEP 518 (declarative dependencies for setup.py), and the competing PEPs 516 & 517 for pluggable build system invocation are designed to better enable that, although Daniel Holth's work with enscons shows that it's possible to bootstrap an existing external build system from setup.py, even without formal support for that at the interoperability specification level. The reason packaging in particular is insistent on the use of documented interoperability standards is that part of the goal is to drop the requirement that build systems for Python projects must be written in Python - they may need or benefit from offering Python bindings, but working with our data formats in other languages should be fully supported.

Requiring pip only to know where distutils puts assets is pretty bad.

This is an area that "just" requires funding and contributions - distlib (pragmatic, not necessarily standards based) and packaging (strictly based on agreed interoperability standards) both exist to serve as pattern extraction libraries for pip, PyPI, setuptools, distutils, sysconfig etc, and make the things they do internally more readily available to other software. However, pursuing this model effectively requires giving up on the idea that distutils will ever be "fixed" - it won't. There are deep structural flaws in the design of its API (which even Greg Ward, its original creator, now considers a bad idea) that can't be addressed without major compatibility breaks, so it makes more sense to enable and pursue the "strangler application" model [1], where we facilitate the adoption of alternative build systems by publishers in a way that's transparent to software consumers, and then base future recommendations for default tooling choices based on emergent community behaviour. The focus for distutils becomes "don't break currently working build processes", while setuptools and other PyPA projects take up the challenge of driving improvements in the capabilities made available to publishers. At the end of that process (which is likely to take years), there shouldn't be any software build activities left using plain distutils - everything that hasn't switched to a different build system entirely will have either switched explicitly to setuptools, or will have setuptools injected into the build process by build environments the same way pip already does. [1] http://www.martinfowler.com/bliki/StranglerApplication.html

Interestingly, on the subject of building extensions (where the has_flag method discussion applies), distutils.ccompiler is the only available tool and happens to not be monkey-patched by setuptools at the moment. Although setuptools re-defines buildext (and tries to get a version from cython first!) So it seems that there no real "right place" within setuptools to monkey-patch distutils' ccompiler.

If I understand the change correctly, the new feature would only be used explicitly, and thus may not require monkeypatching at all. Instead, it could just require importing buildext from setuptools rather than distutils if someone wants the new behaviour on older Python versions. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia