[Distutils] Comments on PEP 426

[Oscar Benjamin]

On 4 September 2013 13:51, Paul Moore <p.f.moore at gmail.com> wrote:
> On 4 September 2013 12:58, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> However, a more significant problem is that the whole idea is based on
>> pure vapourware. That ideal "it's just like setuptools, but with a
>> more elegant implementation that could be used to replace distutils in
>> Python 3.4" library *doesn't exist*, and I have no desire to wait
>> around in the (likely vain) hope of somebody stepping up to create it.
>
> The problem with not even defining an interface is that there is no
> upgrade path. Users use setuptools/pip or wait for the new solution.
> Nobody can write their own replacement for setuptools (bento, for
> example) and be sure it will integrate with pip.
>
> It's a longer term issue for certain, but it *is* important. If we
> don't get away from the implementation defining the behaviour, we'll
> just perpetuate the problem of nobody being able to innovate because
> everything needs to implement most of setuptools.

It doesn't need to be "just like setuptools". Distutils could be made
to be "just like distutils" but with the addition of the *minimum*
that is required to make it work in the new packaging system. Then a
vanilla distutils setup.py can gain the relevant new commands without
the package author needing to put setuptools in the setup.py and
without the user needing to install setuptools and wheel.

Distutils is already just like setuptools for a subset of what
setuptools does. It could be made just like setuptools for a slightly
larger subset that also includes the features deemed to be necessary
in the future. Then pip's setuptools monkey-patching could just phase
out along with the older Python versions where it is necessary.

Projects using Cython often use Cython's distutils extension which
isn't directly compatible with many setuptools commands (although
setuptools-cython apparently tries to address this). These projects
could really benefit from wheel support but they won't get it from
setuptools. It could be added to Cython's distutils extension but then
that doesn't help the packages with non-Cython C extensions which
often don't use setuptools. I think a lot of projects would benefit
from bdist_wheel being added to distutils even if it's only there for
future Python versions.

Until the "minimum that is required" of setup.py and the setup
function has been identified I don't see how you can rule out the
possibility of bringing distutils up to that specification.

>> Instead, I think the far more pragmatic option at this point is to
>> just tell people "your setup.py must run correctly with setuptools
>> imported in that process. If it doesn't, it is your setup.py is
>> broken, not the build tool that imported setuptools, or setuptools
>> itself for monkey-patching distutils".
>
> The big question here, I suppose is do any such projects exist?
> There's a lot of nervousness (I hesitate to use the term FUD) about
> "how will projects that don't work with setuptools react?" but no
> actual evidence of such projects existing. I believe that cx_Oracle
> had an issue in the past. I must try to test that out again and report
> it to them if so. Maybe MAL's projects are another potential test
> case. And maybe numpy. It's hard to be sure, because projects in this
> category are likely the more complex ones, and as a result are
> probably also pretty hard to build (and consequently to test...)

Numpy already has logic to try and do the right thing with or without
setuptools and with different versions of setuptools. If you look
here:
https://github.com/numpy/numpy/blob/master/numpy/distutils/core.py#L6

At the moment it looks like:

if 'setuptools' in sys.modules:
    have_setuptools = True
    from setuptools import setup as old_setup
    # easy_install imports math, it may be picked up from cwd
    from setuptools.command import easy_install
    try:
        # very old versions of setuptools don't have this
        from setuptools.command import bdist_egg
    except ImportError:
        have_setuptools = False
else:
    from distutils.core import setup as old_setup
    have_setuptools = False

Presumably the next step is to add:

try:
    from wheel.bdist_wheel import bdist_wheel
except ImportError:
    have_wheel = False
else:
    have_wheel = True

followed by:

if have_wheel:
    numpy_cmdclass['bdist_wheel'] = bdist_wheel

It looks a bit of a mess but it's worth bearing in mind that the numpy
folks will basically do whatever is required to make all of this stuff
work (not that it's okay to antagonise them with arbitrary changes).

The more relevant concern is how any of this affects smaller and less
well-maintained projects. Setuptools addresses various problems in
distutils for pure Python projects but AFAIK it doesn't make it any
easier to build extension modules. Numpy has all the logic above
precisely because they're trying to emulate the monkey-patching
behaviour of setuptools. However smaller projects will often just use
distutils (perhaps with bespoke monkey-patches or with something like
Cython.Distutils).


Oscar