[Distutils] Comments on PEP 426

[Donald Stufft]

On Sep 4, 2013, at 10:30 AM, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote:

> On 4 September 2013 13:51, Paul Moore <p.f.moore at gmail.com> wrote:
>> On 4 September 2013 12:58, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> 
>>> However, a more significant problem is that the whole idea is based on
>>> pure vapourware. That ideal "it's just like setuptools, but with a
>>> more elegant implementation that could be used to replace distutils in
>>> Python 3.4" library *doesn't exist*, and I have no desire to wait
>>> around in the (likely vain) hope of somebody stepping up to create it.
>> 
>> The problem with not even defining an interface is that there is no
>> upgrade path. Users use setuptools/pip or wait for the new solution.
>> Nobody can write their own replacement for setuptools (bento, for
>> example) and be sure it will integrate with pip.
>> 
>> It's a longer term issue for certain, but it *is* important. If we
>> don't get away from the implementation defining the behaviour, we'll
>> just perpetuate the problem of nobody being able to innovate because
>> everything needs to implement most of setuptools.
> 
> It doesn't need to be "just like setuptools". Distutils could be made
> to be "just like distutils" but with the addition of the *minimum*
> that is required to make it work in the new packaging system. Then a
> vanilla distutils setup.py can gain the relevant new commands without
> the package author needing to put setuptools in the setup.py and
> without the user needing to install setuptools and wheel.
> 
> Distutils is already just like setuptools for a subset of what
> setuptools does. It could be made just like setuptools for a slightly
> larger subset that also includes the features deemed to be necessary
> in the future. Then pip's setuptools monkey-patching could just phase
> out along with the older Python versions where it is necessary.
> 
> Projects using Cython often use Cython's distutils extension which
> isn't directly compatible with many setuptools commands (although
> setuptools-cython apparently tries to address this). These projects
> could really benefit from wheel support but they won't get it from
> setuptools. It could be added to Cython's distutils extension but then
> that doesn't help the packages with non-Cython C extensions which
> often don't use setuptools. I think a lot of projects would benefit
> from bdist_wheel being added to distutils even if it's only there for
> future Python versions.
> 
> Until the "minimum that is required" of setup.py and the setup
> function has been identified I don't see how you can rule out the
> possibility of bringing distutils up to that specification.

Even if distutils is up to that specification there are 5 (likely 6) legacy
versions that are not. If users want to use distutils they have to
completely restrict themselves (and all of their users) to only versions
of python 3.4 (likely 3.5) or higher. That's unreasonable for most
projects.

> 
>>> Instead, I think the far more pragmatic option at this point is to
>>> just tell people "your setup.py must run correctly with setuptools
>>> imported in that process. If it doesn't, it is your setup.py is
>>> broken, not the build tool that imported setuptools, or setuptools
>>> itself for monkey-patching distutils".
>> 
>> The big question here, I suppose is do any such projects exist?
>> There's a lot of nervousness (I hesitate to use the term FUD) about
>> "how will projects that don't work with setuptools react?" but no
>> actual evidence of such projects existing. I believe that cx_Oracle
>> had an issue in the past. I must try to test that out again and report
>> it to them if so. Maybe MAL's projects are another potential test
>> case. And maybe numpy. It's hard to be sure, because projects in this
>> category are likely the more complex ones, and as a result are
>> probably also pretty hard to build (and consequently to test...)
> 
> Numpy already has logic to try and do the right thing with or without
> setuptools and with different versions of setuptools. If you look
> here:
> https://github.com/numpy/numpy/blob/master/numpy/distutils/core.py#L6
> 
> At the moment it looks like:
> 
> if 'setuptools' in sys.modules:
>    have_setuptools = True
>    from setuptools import setup as old_setup
>    # easy_install imports math, it may be picked up from cwd
>    from setuptools.command import easy_install
>    try:
>        # very old versions of setuptools don't have this
>        from setuptools.command import bdist_egg
>    except ImportError:
>        have_setuptools = False
> else:
>    from distutils.core import setup as old_setup
>    have_setuptools = False
> 
> Presumably the next step is to add:
> 
> try:
>    from wheel.bdist_wheel import bdist_wheel
> except ImportError:
>    have_wheel = False
> else:
>    have_wheel = True
> 
> followed by:
> 
> if have_wheel:
>    numpy_cmdclass['bdist_wheel'] = bdist_wheel
> 
> It looks a bit of a mess but it's worth bearing in mind that the numpy
> folks will basically do whatever is required to make all of this stuff
> work (not that it's okay to antagonise them with arbitrary changes).
> 
> The more relevant concern is how any of this affects smaller and less
> well-maintained projects. Setuptools addresses various problems in
> distutils for pure Python projects but AFAIK it doesn't make it any
> easier to build extension modules. Numpy has all the logic above
> precisely because they're trying to emulate the monkey-patching
> behaviour of setuptools. However smaller projects will often just use
> distutils (perhaps with bespoke monkey-patches or with something like
> Cython.Distutils).

I don't think setuptools makes it any harder to build extension modules
either. In fact afaik it enables via the setup_requires and entry points to
override the entire build/install command without needing to mess around
with handling command classes.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130904/3800b3e5/attachment.sig>