What is the official position on distutils?
The discussion of Sylvain's proposed changes to distutils suggests that there isn't a clear-cut agreement or position of this SIG -- and thus Python -- on changes to distutils, its future, etc. Is there an official position I'm not aware of? If not, could we get one so we know how to handle any more proposed changes to distutils?
For me personally (and to start this conversation if necessary), I view distutils as the package in the stdlib that handles building extension modules in the stdlib for Python. That view means that if something doesn't service that explicit goal then it shouldn't go into distutils. Now that we are moving down the road towards making the build step configurable I'm fine with saying that distutils is not expected to work for people other than Python itself and others can use setuptools, enscons, or other projects while we continue to improve the situation to where the build system is just something you specify in pyproject.toml to generate your wheel. IOW distutils is only exposed publicly because Python doesn't hide anything, but making it useful for the general case of direct usage by people is a non-goal of the package.
My sense of the current position is:
- distutils exists for both Python and the broader ecosystem to build both extension modules and package distributions - we recognise that its able to do anything, but many folk find it hard to work with - or are already invested in other build tools - we're afraid of changing it in ways that cause incompatibilities with e.g. the numpy eco systems heavy customisations, and we're not sure we can recognise whether a given change is one of those - and we're recommending that everyone move away from directly using distutils to using setuptoools - but setuptools *is* distutils in many many ways, so this doesn't loosen our dependency and stability considerations around distutils
So basically the normal 'module goes to the stdlib to freeze' effect.
I haven't read about Sylvain's proposed changes yet - ETHINGSTODO :(.
I can happily agree with a position that making distutils directly usable is a non-goal... but insofar as making it more usable makes setuptools more usable, that would be a goal :/.
-Rob
On 29 August 2016 at 05:05, Brett Cannon brett@python.org wrote:
The discussion of Sylvain's proposed changes to distutils suggests that there isn't a clear-cut agreement or position of this SIG -- and thus Python -- on changes to distutils, its future, etc. Is there an official position I'm not aware of? If not, could we get one so we know how to handle any more proposed changes to distutils?
For me personally (and to start this conversation if necessary), I view distutils as the package in the stdlib that handles building extension modules in the stdlib for Python. That view means that if something doesn't service that explicit goal then it shouldn't go into distutils. Now that we are moving down the road towards making the build step configurable I'm fine with saying that distutils is not expected to work for people other than Python itself and others can use setuptools, enscons, or other projects while we continue to improve the situation to where the build system is just something you specify in pyproject.toml to generate your wheel. IOW distutils is only exposed publicly because Python doesn't hide anything, but making it useful for the general case of direct usage by people is a non-goal of the package.
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 28 August 2016 at 18:05, Brett Cannon brett@python.org wrote:
The discussion of Sylvain's proposed changes to distutils suggests that there isn't a clear-cut agreement or position of this SIG -- and thus Python -- on changes to distutils, its future, etc. Is there an official position I'm not aware of? If not, could we get one so we know how to handle any more proposed changes to distutils?
For me personally (and to start this conversation if necessary), I view distutils as the package in the stdlib that handles building extension modules in the stdlib for Python. That view means that if something doesn't service that explicit goal then it shouldn't go into distutils. Now that we are moving down the road towards making the build step configurable I'm fine with saying that distutils is not expected to work for people other than Python itself and others can use setuptools, enscons, or other projects while we continue to improve the situation to where the build system is just something you specify in pyproject.toml to generate your wheel. IOW distutils is only exposed publicly because Python doesn't hide anything, but making it useful for the general case of direct usage by people is a non-goal of the package.
FWIW, my view is:
* distutils handles building of Python packages, both in the stdlib and outside * in practice, setuptools is almost always used with distutils, and any proposed change to distutils could be made to setuptools just as easily * patches for distutils typically suffer from a lack of maintainers motivated to work on them * there are a few developers with a strong investment in distutils, who have code that is deeply entwined with the internals of distutils, and changes to distutils come with a high risk of causing issues for those people * these developers *don't* use setuptools, so changes in setuptools don't tend to suffer from the same level of backward compatibility constraints * implementing changes in setuptools is *far* easier than doing so in distutils, for the above reasons * changes in setuptools can be used with any version of Python, distutils changes are only available to people with no need to support anything but the latest version (which in practice is a vanishingly small group) * as a result, there would have to be an *extremely* compelling reason to make a change in distutils rather than in setuptools - sufficient to justify the additional risk, the extra developer effort needed, and the fact that any such change is only going to benefit users of newer versions of Python
The reality is simply that distutils is in a somewhat unhappy place, with too few interested maintainers and serious compatibility problems to deal with, but that's not exactly news to anyone. I don't think that a formal statement will help much. Past history has shown that making pronouncements like "distutils is frozen" has not helped anyone - so we're just trying to work with the status quo.
With regard to Sylvain's proposed changes, I haven't really seen any arguments justifying making the change in distutils rather than in setuptools - the last suggestion I saw was to make the change in *both* places, but I don't see any particular reason (beyond abstract "it makes more sense to have the change in distutils" arguments) for not just targeting setuptools. I may be missing something (as I'm not personally in a position where I'd be comfortable reviewing the change) but it seems to me that it's a bit of a non-issue - just implement in setuptools and move on. The rhetoric about "this means that distutils is dead" doesn't feel particularly helpful.
Paul
Distutils seems to be the de-facto standard for building extension modules for Python and it is used for most of the third-party extensions out there. I don’t think that it is reasonable to declare that it is now only meant for Python itself. I actually see a contradiction in pointing out some lack of involvement in the library to oppose well needed improvements, especially for a tool that has so much adoption.
As we discussed earlier, even though it is not a concern with C, checking for availability of a compiler flag becomes crucial when building extensions with C++ since new flavors of the language emerge every couple of years now. It is important to be able to output meaningful error messages when the compiler does not support C++[11/14/17] features if they are needed for a given extension. This is a new aspect of the landscape in this area.
Finally, adding this method is a very straightforward change. `has_flag` simply comes aside `has_function` as a method of ccompiler. I don't see a more natural place for it. It would be a very weird design decision in my opinion to not add it there, and instead to add it to distutils.ccompiler by monkeypatching it in setuptools.
Thanks,
Sylvain
On Mon, Aug 29, 2016 at 12:36 AM, Paul Moore p.f.moore@gmail.com wrote:
On 28 August 2016 at 18:05, Brett Cannon brett@python.org wrote:
The discussion of Sylvain's proposed changes to distutils suggests that there isn't a clear-cut agreement or position of this SIG -- and thus
Python
-- on changes to distutils, its future, etc. Is there an official
position
I'm not aware of? If not, could we get one so we know how to handle any
more
proposed changes to distutils?
For me personally (and to start this conversation if necessary), I view distutils as the package in the stdlib that handles building extension modules in the stdlib for Python. That view means that if something
doesn't
service that explicit goal then it shouldn't go into distutils. Now that
we
are moving down the road towards making the build step configurable I'm
fine
with saying that distutils is not expected to work for people other than Python itself and others can use setuptools, enscons, or other projects while we continue to improve the situation to where the build system is
just
something you specify in pyproject.toml to generate your wheel. IOW distutils is only exposed publicly because Python doesn't hide anything,
but
making it useful for the general case of direct usage by people is a non-goal of the package.
FWIW, my view is:
- distutils handles building of Python packages, both in the stdlib and
outside
- in practice, setuptools is almost always used with distutils, and
any proposed change to distutils could be made to setuptools just as easily
- patches for distutils typically suffer from a lack of maintainers
motivated to work on them
- there are a few developers with a strong investment in distutils,
who have code that is deeply entwined with the internals of distutils, and changes to distutils come with a high risk of causing issues for those people
- these developers *don't* use setuptools, so changes in setuptools
don't tend to suffer from the same level of backward compatibility constraints
- implementing changes in setuptools is *far* easier than doing so in
distutils, for the above reasons
- changes in setuptools can be used with any version of Python,
distutils changes are only available to people with no need to support anything but the latest version (which in practice is a vanishingly small group)
- as a result, there would have to be an *extremely* compelling reason
to make a change in distutils rather than in setuptools - sufficient to justify the additional risk, the extra developer effort needed, and the fact that any such change is only going to benefit users of newer versions of Python
The reality is simply that distutils is in a somewhat unhappy place, with too few interested maintainers and serious compatibility problems to deal with, but that's not exactly news to anyone. I don't think that a formal statement will help much. Past history has shown that making pronouncements like "distutils is frozen" has not helped anyone
- so we're just trying to work with the status quo.
With regard to Sylvain's proposed changes, I haven't really seen any arguments justifying making the change in distutils rather than in setuptools - the last suggestion I saw was to make the change in *both* places, but I don't see any particular reason (beyond abstract "it makes more sense to have the change in distutils" arguments) for not just targeting setuptools. I may be missing something (as I'm not personally in a position where I'd be comfortable reviewing the change) but it seems to me that it's a bit of a non-issue - just implement in setuptools and move on. The rhetoric about "this means that distutils is dead" doesn't feel particularly helpful.
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Sun, 28 Aug 2016 at 16:55 Sylvain Corlay sylvain.corlay@gmail.com wrote:
Distutils seems to be the de-facto standard for building extension modules for Python and it is used for most of the third-party extensions out there. I don’t think that it is reasonable to declare that it is now only meant for Python itself.
Sorry, I didn't mean to suggest distutils was suddenly going to change or disappear, but I did want a direction/focus for what we want distutils to become long-term as we have this discussion of what to do with distutils any time a change is proposed. At this point distutils is practically a black box of code as no one wants to touch out of fear of breaking the world and that's not sustainable.
I actually see a contradiction in pointing out some lack of involvement in the library to oppose well needed improvements, especially for a tool that has so much adoption.
It's not a contradiction because new features != maintenance. And the usage of distutils makes its lack of maintenance even more scary. I guess my key point from what I said in my previous email is that we shouldn't perpetually ignore the fact that distutils is not to a level of support or quality that any of us are happy with and yet so many people rely on.
For me this means we should prioritize getting our build API defined and sdists standardized so that distutils (or a replacement) can be in the stdlib to build Python itself only, and after that you get your build tool from PyPI. And if that means setuptools swallows/subsumes distutils then that's fine with me, but my key point is we are not maintaining distutils in a way I think any of us would choose to now and so I'm trying to prod this question to start thinking about how to fix the maintenance problem long-term.
As we discussed earlier, even though it is not a concern with C, checking for availability of a compiler flag becomes crucial when building extensions with C++ since new flavors of the language emerge every couple of years now. It is important to be able to output meaningful error messages when the compiler does not support C++[11/14/17] features if they are needed for a given extension. This is a new aspect of the landscape in this area.
Finally, adding this method is a very straightforward change. `has_flag` simply comes aside `has_function` as a method of ccompiler. I don't see a more natural place for it. It would be a very weird design decision in my opinion to not add it there, and instead to add it to distutils.ccompiler by monkeypatching it in setuptools.
Honestly I have no comment on your feature, Sylvain, and I'm sorry your proposal happens to be the catalyst to this discussion. I'm just trying to get a general alignment from the PyPA group as the "distutils problem" comes up and has the same points made every time with no general decision on how to handle it long-term.
-Brett
Thanks,
Sylvain
On Mon, Aug 29, 2016 at 12:36 AM, Paul Moore p.f.moore@gmail.com wrote:
On 28 August 2016 at 18:05, Brett Cannon brett@python.org wrote:
The discussion of Sylvain's proposed changes to distutils suggests that there isn't a clear-cut agreement or position of this SIG -- and thus
Python
-- on changes to distutils, its future, etc. Is there an official
position
I'm not aware of? If not, could we get one so we know how to handle any
more
proposed changes to distutils?
For me personally (and to start this conversation if necessary), I view distutils as the package in the stdlib that handles building extension modules in the stdlib for Python. That view means that if something
doesn't
service that explicit goal then it shouldn't go into distutils. Now
that we
are moving down the road towards making the build step configurable I'm
fine
with saying that distutils is not expected to work for people other than Python itself and others can use setuptools, enscons, or other projects while we continue to improve the situation to where the build system is
just
something you specify in pyproject.toml to generate your wheel. IOW distutils is only exposed publicly because Python doesn't hide
anything, but
making it useful for the general case of direct usage by people is a non-goal of the package.
FWIW, my view is:
- distutils handles building of Python packages, both in the stdlib and
outside
- in practice, setuptools is almost always used with distutils, and
any proposed change to distutils could be made to setuptools just as easily
- patches for distutils typically suffer from a lack of maintainers
motivated to work on them
- there are a few developers with a strong investment in distutils,
who have code that is deeply entwined with the internals of distutils, and changes to distutils come with a high risk of causing issues for those people
- these developers *don't* use setuptools, so changes in setuptools
don't tend to suffer from the same level of backward compatibility constraints
- implementing changes in setuptools is *far* easier than doing so in
distutils, for the above reasons
- changes in setuptools can be used with any version of Python,
distutils changes are only available to people with no need to support anything but the latest version (which in practice is a vanishingly small group)
- as a result, there would have to be an *extremely* compelling reason
to make a change in distutils rather than in setuptools - sufficient to justify the additional risk, the extra developer effort needed, and the fact that any such change is only going to benefit users of newer versions of Python
The reality is simply that distutils is in a somewhat unhappy place, with too few interested maintainers and serious compatibility problems to deal with, but that's not exactly news to anyone. I don't think that a formal statement will help much. Past history has shown that making pronouncements like "distutils is frozen" has not helped anyone
- so we're just trying to work with the status quo.
With regard to Sylvain's proposed changes, I haven't really seen any arguments justifying making the change in distutils rather than in setuptools - the last suggestion I saw was to make the change in *both* places, but I don't see any particular reason (beyond abstract "it makes more sense to have the change in distutils" arguments) for not just targeting setuptools. I may be missing something (as I'm not personally in a position where I'd be comfortable reviewing the change) but it seems to me that it's a bit of a non-issue - just implement in setuptools and move on. The rhetoric about "this means that distutils is dead" doesn't feel particularly helpful.
Paul
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Hi Brett,
On Mon, Aug 29, 2016 at 9:52 PM, Brett Cannon brett@python.org wrote:
As we discussed earlier, even though it is not a concern with C, checking for availability of a compiler flag becomes crucial when building extensions with C++ since new flavors of the language emerge every couple of years now. It is important to be able to output meaningful error messages when the compiler does not support C++[11/14/17] features if they are needed for a given extension. This is a new aspect of the landscape in this area.
Finally, adding this method is a very straightforward change. `has_flag` simply comes aside `has_function` as a method of ccompiler. I don't see a more natural place for it. It would be a very weird design decision in my opinion to not add it there, and instead to add it to distutils.ccompiler by monkeypatching it in setuptools.
Honestly I have no comment on your feature, Sylvain, and I'm sorry your proposal happens to be the catalyst to this discussion. I'm just trying to get a general alignment from the PyPA group as the "distutils problem" comes up and has the same points made every time with no general decision on how to handle it long-term.
-Brett
My point here was that I don't think that the proposed feature has much to do with the concerns that were raised about distutils in general, unless it is decided that incremental improvements to the library even backward compatible will not be accepted anymore.
Sylvain
On 2 September 2016 at 09:58, Sylvain Corlay sylvain.corlay@gmail.com wrote:
My point here was that I don't think that the proposed feature has much to do with the concerns that were raised about distutils in general, unless it is decided that incremental improvements to the library even backward compatible will not be accepted anymore.
Agreed. I think your feature is only stalled for distutils by the lack of people sufficiently comfortable with the code to apply it. The suggestions to add it to setuptools are more in the way of practical advice on how to make the feature available, rather than expressions of a policy that "we don't make changes like that in the stdlib".
Paul
On 2 September 2016 at 19:28, Paul Moore p.f.moore@gmail.com wrote:
On 2 September 2016 at 09:58, Sylvain Corlay sylvain.corlay@gmail.com wrote:
My point here was that I don't think that the proposed feature has much to do with the concerns that were raised about distutils in general, unless it is decided that incremental improvements to the library even backward compatible will not be accepted anymore.
Agreed. I think your feature is only stalled for distutils by the lack of people sufficiently comfortable with the code to apply it. The suggestions to add it to setuptools are more in the way of practical advice on how to make the feature available, rather than expressions of a policy that "we don't make changes like that in the stdlib".
However, one of the other consequences of the status quo is that if Jason's comfortable with a change for setuptools, there's very rarely going to be anyone that will argue with him if he also considers it a suitable addition to the next version of distutils :)
Since Jason's primary involvement in distutils-sig & PyPA is as the lead setuptools maintainer, it's easy for folks to be unaware of the fact that he's a distutils maintainer as well.
So perhaps that's what we should adopt as the official distutils-sig policy? Any proposed distutils changes should *always* go into setuptools, as that way they're available for all currently supported Python versions, and then it's up to the setuptools project to escalate changes or change proposals for stdlib inclusion when they consider that an appropriate step.
Cheers, Nick.
On Fri, 2 Sep 2016 at 22:06 Nick Coghlan ncoghlan@gmail.com wrote:
On 2 September 2016 at 19:28, Paul Moore p.f.moore@gmail.com wrote:
On 2 September 2016 at 09:58, Sylvain Corlay sylvain.corlay@gmail.com
wrote:
My point here was that I don't think that the proposed feature has much
to
do with the concerns that were raised about distutils in general,
unless it
is decided that incremental improvements to the library even backward compatible will not be accepted anymore.
Agreed. I think your feature is only stalled for distutils by the lack of people sufficiently comfortable with the code to apply it. The suggestions to add it to setuptools are more in the way of practical advice on how to make the feature available, rather than expressions of a policy that "we don't make changes like that in the stdlib".
However, one of the other consequences of the status quo is that if Jason's comfortable with a change for setuptools, there's very rarely going to be anyone that will argue with him if he also considers it a suitable addition to the next version of distutils :)
Since Jason's primary involvement in distutils-sig & PyPA is as the lead setuptools maintainer, it's easy for folks to be unaware of the fact that he's a distutils maintainer as well.
So perhaps that's what we should adopt as the official distutils-sig policy? Any proposed distutils changes should *always* go into setuptools, as that way they're available for all currently supported Python versions, and then it's up to the setuptools project to escalate changes or change proposals for stdlib inclusion when they consider that an appropriate step.
If Jason is up for the responsibility that seems like a reasonable approach to take. It also helps test out features in setuptools first before upstreaming it.
Hi Brett,
On Sat, Sep 3, 2016 at 8:05 PM, Brett Cannon brett@python.org wrote:
If Jason is up for the responsibility that seems like a reasonable approach to take. It also helps test out features in setuptools first before upstreaming it.
How do you see `has_flag` get into setuptools? By monkey-patching distutils' ccompiler to have it aside of `has_function` when setuptools is imported?
I find really weird the idea of adding this in a convoluted fashion instead of allowing incremental improvement of distutils.
Sylvain
On 4 September 2016 at 06:44, Sylvain Corlay sylvain.corlay@gmail.com wrote:
Hi Brett,
On Sat, Sep 3, 2016 at 8:05 PM, Brett Cannon brett@python.org wrote:
If Jason is up for the responsibility that seems like a reasonable approach to take. It also helps test out features in setuptools first before upstreaming it.
How do you see `has_flag` get into setuptools? By monkey-patching distutils' ccompiler to have it aside of `has_function` when setuptools is imported?
I find really weird the idea of adding this in a convoluted fashion instead of allowing incremental improvement of distutils.
The change to distutils would still be a plain patch to distutils, it would just be accepted at the API design level in setuptools first.
The problem you're running into right now isn't a technical one - it's that there isn't anyone that currently feels like they have sufficient design authority over the distutils API to accept your proposal, hence Brett starting this thread to address that underlying recurring question, rather than the specifics of your change.
Jason *definitely* has that design authority over setuptools though, and will be tasked with making any API additions available on older versions of Python via setuptools regardless of what policy we adopt for distutils maintenance, so if he's amenable to the idea, it makes sense to me to invert the order we ask those questions: add it to setuptools first, and then add things to distutils where we feel they're sufficiently stable to not need the benefit of the faster setuptools update cycle.
Cheers, Nick.
"add it to setuptools first, and then add things to distutils where we feel they're sufficiently stable to not need the benefit of the faster setuptools update cycle"
I nominate this as the official policy on API changes for distutils, with regular maintenance mode policies applying to anything else.
Top-posted from my Windows Phone
-----Original Message----- From: "Nick Coghlan" ncoghlan@gmail.com Sent: 9/4/2016 2:19 To: "Sylvain Corlay" sylvain.corlay@gmail.com Cc: "distutils sig" distutils-sig@python.org Subject: Re: [Distutils] What is the official position on distutils?
On 4 September 2016 at 06:44, Sylvain Corlay sylvain.corlay@gmail.com wrote:
Hi Brett,
On Sat, Sep 3, 2016 at 8:05 PM, Brett Cannon brett@python.org wrote:
If Jason is up for the responsibility that seems like a reasonable approach to take. It also helps test out features in setuptools first before upstreaming it.
How do you see `has_flag` get into setuptools? By monkey-patching distutils' ccompiler to have it aside of `has_function` when setuptools is imported?
I find really weird the idea of adding this in a convoluted fashion instead of allowing incremental improvement of distutils.
The change to distutils would still be a plain patch to distutils, it would just be accepted at the API design level in setuptools first.
The problem you're running into right now isn't a technical one - it's that there isn't anyone that currently feels like they have sufficient design authority over the distutils API to accept your proposal, hence Brett starting this thread to address that underlying recurring question, rather than the specifics of your change.
Jason *definitely* has that design authority over setuptools though, and will be tasked with making any API additions available on older versions of Python via setuptools regardless of what policy we adopt for distutils maintenance, so if he's amenable to the idea, it makes sense to me to invert the order we ask those questions: add it to setuptools first, and then add things to distutils where we feel they're sufficiently stable to not need the benefit of the faster setuptools update cycle.
Cheers, Nick.
+1
On Sun, Sep 4, 2016 at 7:02 AM, Steve Dower steve.dower@python.org wrote:
"add it to setuptools first, and then add things to distutils where we feel they're sufficiently stable to not need the benefit of the faster setuptools update cycle"
I nominate this as the official policy on API changes for distutils, with regular maintenance mode policies applying to anything else.
Top-posted from my Windows Phone
From: Nick Coghlan ncoghlan@gmail.com Sent: 9/4/2016 2:19 To: Sylvain Corlay sylvain.corlay@gmail.com Cc: distutils sig distutils-sig@python.org Subject: Re: [Distutils] What is the official position on distutils?
On 4 September 2016 at 06:44, Sylvain Corlay sylvain.corlay@gmail.com wrote:
Hi Brett,
On Sat, Sep 3, 2016 at 8:05 PM, Brett Cannon brett@python.org wrote:
If Jason is up for the responsibility that seems like a reasonable approach to take. It also helps test out features in setuptools first
before
upstreaming it.
How do you see `has_flag` get into setuptools? By monkey-patching
distutils'
ccompiler to have it aside of `has_function` when setuptools is imported?
I find really weird the idea of adding this in a convoluted fashion
instead
of allowing incremental improvement of distutils.
The change to distutils would still be a plain patch to distutils, it would just be accepted at the API design level in setuptools first.
The problem you're running into right now isn't a technical one - it's that there isn't anyone that currently feels like they have sufficient design authority over the distutils API to accept your proposal, hence Brett starting this thread to address that underlying recurring question, rather than the specifics of your change.
Jason *definitely* has that design authority over setuptools though, and will be tasked with making any API additions available on older versions of Python via setuptools regardless of what policy we adopt for distutils maintenance, so if he's amenable to the idea, it makes sense to me to invert the order we ask those questions: add it to setuptools first, and then add things to distutils where we feel they're sufficiently stable to not need the benefit of the faster setuptools update cycle.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Sat, Sep 3, 2016 at 5:06 PM, Nick Coghlan ncoghlan@gmail.com wrote:
On 2 September 2016 at 19:28, Paul Moore p.f.moore@gmail.com wrote:
On 2 September 2016 at 09:58, Sylvain Corlay sylvain.corlay@gmail.com
wrote:
My point here was that I don't think that the proposed feature has much
to
do with the concerns that were raised about distutils in general,
unless it
is decided that incremental improvements to the library even backward compatible will not be accepted anymore.
Agreed. I think your feature is only stalled for distutils by the lack of people sufficiently comfortable with the code to apply it. The suggestions to add it to setuptools are more in the way of practical advice on how to make the feature available, rather than expressions of a policy that "we don't make changes like that in the stdlib".
However, one of the other consequences of the status quo is that if Jason's comfortable with a change for setuptools, there's very rarely going to be anyone that will argue with him if he also considers it a suitable addition to the next version of distutils :)
Since Jason's primary involvement in distutils-sig & PyPA is as the lead setuptools maintainer, it's easy for folks to be unaware of the fact that he's a distutils maintainer as well.
So perhaps that's what we should adopt as the official distutils-sig policy? Any proposed distutils changes should *always* go into setuptools, as that way they're available for all currently supported Python versions,
and better maintained, and easier to fix if there's bugs, etc.
and then it's up to the setuptools project to escalate changes or change proposals for stdlib inclusion when they consider that an appropriate step.
+1. clear and pragmatic policy.
Ralf
On Sun, 28 Aug 2016 23:36:43 +0100 Paul Moore p.f.moore@gmail.com wrote:
- as a result, there would have to be an *extremely* compelling reason
to make a change in distutils rather than in setuptools - sufficient to justify the additional risk, the extra developer effort needed, and the fact that any such change is only going to benefit users of newer versions of Python
There is a problem with this: distutils and setuptools don't document or specify their internal APIs. Yet if you want to do something advanced, you have to hook into the command classes and either call their APIs or extend them.
Using distutils, this is already tedious, since you have to go read its source code and try to figure out what happens (it's rarely obvious).
Using setuptools, this is even worse, though, because setuptools builds on top of distutils by modifying the command classes, and so you have to figure out the effect that monkeypatching has on the overall behaviour (in addition to figure out what happens in distutils).
I don't know if that's bothersome enough to compound the fact that changes made in setuptools can benefit all Python versions, but I think it's useful to keep it in mind.
(that lack of programmability is really distutils' failure, of course, and setuptools inherited that failure by inheriting distutils' architecture.)
Regards
Antoine.
On 30 August 2016 at 12:14, Antoine Pitrou solipsis@pitrou.net wrote:
On Sun, 28 Aug 2016 23:36:43 +0100 Paul Moore p.f.moore@gmail.com wrote:
- as a result, there would have to be an *extremely* compelling reason
to make a change in distutils rather than in setuptools - sufficient to justify the additional risk, the extra developer effort needed, and the fact that any such change is only going to benefit users of newer versions of Python
There is a problem with this: distutils and setuptools don't document or specify their internal APIs. Yet if you want to do something advanced, you have to hook into the command classes and either call their APIs or extend them.
Using distutils, this is already tedious, since you have to go read its source code and try to figure out what happens (it's rarely obvious).
Using setuptools, this is even worse, though, because setuptools builds on top of distutils by modifying the command classes, and so you have to figure out the effect that monkeypatching has on the overall behaviour (in addition to figure out what happens in distutils).
I don't know if that's bothersome enough to compound the fact that changes made in setuptools can benefit all Python versions, but I think it's useful to keep it in mind.
(that lack of programmability is really distutils' failure, of course, and setuptools inherited that failure by inheriting distutils' architecture.)
From what I know, anyone who is actually hacking into the internal
APIs to that level tends to use only distutils (probably because as you say, setuptools is even worse) and therefore falls into my category of "people affected by distutils changes who won't mind if a change is made to setuptools". Admittedly, that's a self-perpetuating situation...
The fact remains that (as far as it's possible to tell) very few people do dive into distutils to that level. At least, not many people seem to have problems caused by the current "leave distutils alone, make changes in setuptools" approach. Ideally, someone would be working on alternative solutions to break the remaining few's tight dependency on distutils, but with the effort they have invested in it, there doesn't seem to be much interest in re-engineering something that works.
Paul
On Tue, 30 Aug 2016 13:34:22 +0100 Paul Moore p.f.moore@gmail.com wrote:
From what I know, anyone who is actually hacking into the internal APIs to that level tends to use only distutils (probably because as you say, setuptools is even worse) and therefore falls into my category of "people affected by distutils changes who won't mind if a change is made to setuptools".
Except when, as you point out, pip injects setuptools nilly-willy when running setup.py?
Regards
Antoine.
On 30 August 2016 at 13:38, Antoine Pitrou solipsis@pitrou.net wrote:
On Tue, 30 Aug 2016 13:34:22 +0100 Paul Moore p.f.moore@gmail.com wrote:
From what I know, anyone who is actually hacking into the internal APIs to that level tends to use only distutils (probably because as you say, setuptools is even worse) and therefore falls into my category of "people affected by distutils changes who won't mind if a change is made to setuptools".
Except when, as you point out, pip injects setuptools nilly-willy when running setup.py?
I'm confused. People who hack into distutils internals can't cope with having setuptools injected (because their hacks conflict with setuptools' hacks). That's my understanding of the situation, but it's based mainly on what I've seen of people saying "Don't break my APIs!!!" when distutils changes have been proposed in the past. The projects involved are already outside of the current pip/setuptools/wheel infrastructure (they have their own install processes, they don't host on PyPI, they are not pip-installable or whatever). So injecting setuptools isn't relevant, because they are outside that particular ecosystem.
What I'm trying to say (badly, it seems - for which I apologise) is that if we change setuptools, only people working in the current ecosystem are affected, and those projects have typically been OK with the level of change going on in setuptools. People outside of the pip/setuptools ecosystem rely on distutils, and appear to be heavier users of the internal distutils APIs. As a result they appear to be typically much more conservative in their requirements (precisely because they had to work with an undocumented internal API). So there's a much higher barrier to changing distutils, and making changes in setuptools instead is less likely to break existing projects (at a cost of making such changes not accessible to non-setuptools projects).
This is largely off-topic by now, though. I wasn't trying to derail the discussion about policy on distutils, so I'll drop this subthread now, with apologies to anyone who found it a distraction from the main point.
Paul
On Tue, 30 Aug 2016 14:44:10 +0100 Paul Moore p.f.moore@gmail.com wrote:
On 30 August 2016 at 13:38, Antoine Pitrou solipsis@pitrou.net wrote:
On Tue, 30 Aug 2016 13:34:22 +0100 Paul Moore p.f.moore@gmail.com wrote:
From what I know, anyone who is actually hacking into the internal APIs to that level tends to use only distutils (probably because as you say, setuptools is even worse) and therefore falls into my category of "people affected by distutils changes who won't mind if a change is made to setuptools".
Except when, as you point out, pip injects setuptools nilly-willy when running setup.py?
I'm confused. People who hack into distutils internals can't cope with having setuptools injected (because their hacks conflict with setuptools' hacks).
I'm not sure what you mean by that. I'm sure there are situations where it works (I can't remember for sure whether it did the last time I did such "hacking").
The thing is, more and more problem expect packages to install with "pip install ." and they report bugs for that (the PyPA's official discours is of course partly responsible for this, since it's insisting so heavily on using pip and discouraging direct calls to setup.py).
And the day people expect all libraries to be available as wheel files, it will probably become worse, because I don't know how to produce wheel files using plain distutils.
Regards
Antoine.
On 30 August 2016 at 23:44, Paul Moore p.f.moore@gmail.com wrote:
On 30 August 2016 at 13:38, Antoine Pitrou solipsis@pitrou.net wrote:
On Tue, 30 Aug 2016 13:34:22 +0100 Paul Moore p.f.moore@gmail.com wrote:
From what I know, anyone who is actually hacking into the internal APIs to that level tends to use only distutils (probably because as you say, setuptools is even worse) and therefore falls into my category of "people affected by distutils changes who won't mind if a change is made to setuptools".
Except when, as you point out, pip injects setuptools nilly-willy when running setup.py?
I'm confused. People who hack into distutils internals can't cope with having setuptools injected (because their hacks conflict with setuptools' hacks). That's my understanding of the situation, but it's based mainly on what I've seen of people saying "Don't break my APIs!!!" when distutils changes have been proposed in the past. The projects involved are already outside of the current pip/setuptools/wheel infrastructure (they have their own install processes, they don't host on PyPI, they are not pip-installable or whatever). So injecting setuptools isn't relevant, because they are outside that particular ecosystem.
No, this isn't the case - if you want to reliably generate wheels (as bdist_wheel doesn't work with plain distutils), and if you want the installdb metadata to always be generated, then you need to use pip and its forced setuptools injection in order to get plain distutils projects to behave the way you want.
This means that other setup.py helper projects like numpy.distutils need to be setuptools-tolerant so they can support folks using pip rather than calling setup.py directly.
That consequence further means that there's isn't substantially lower risk in only changing setuptools without changing distutils - in fact, by introducing a further unhelpful discrepancy between the two, it increases the risk of code that works with plain distutils on 3.6+ breaking when setuptools is injected.
Cheers, Nick.
Sylvain's feature seems totally harmless to me. If he wants to detect features in Python 3.6 and above, more power to him. I love it.
If only it was sufficient to "read" the code for distutils. You will be single-stepping it in a debugger, I guarantee it.
So let's imagine you wanted to create wheels with distutils without installing setuptols. Technically you could write a tool do to that without changing distutils by processing the output of "setup.py install" and converting distutils' PKG-INFO to METADATA. Please note, the resulting package has zero dependency information, dependencies are a setuptools-only feature. It makes me sad to think about how completely drab and pointless this would be. No offense meant.
Distutils does provide some useful features. If you subclass the build_ext command and remove the "execute the compiler" step, you can use it to get the compiler flags expected for C extensions. It also knows how to find the right version of MSVC on Windows. Very handy:
class no_build_ext(build_ext): def build_extension(self, ext): def noop_spawn(*args): pass self.compiler.spawn = noop_spawn build_ext.build_extension(self, ext)
IMO a useful, forward-looking refactor of distutils would take these kinds of features "one weird trick to link Python modules" of which there are several, and expose them independently of the command class system. Then you could feed it into CMake or whatever and build some extensions. No I don't know exactly how that would work.
It's also unimaginable that we would ever stop needing a working version of distutils & setuptools, given that we have hundreds of thousands of existing distributions that use it.
Daniel
On 29 August 2016 at 03:05, Brett Cannon brett@python.org wrote:
The discussion of Sylvain's proposed changes to distutils suggests that there isn't a clear-cut agreement or position of this SIG -- and thus Python -- on changes to distutils, its future, etc. Is there an official position I'm not aware of? If not, could we get one so we know how to handle any more proposed changes to distutils?
I agree it's not clear, and we should be more explicit about it.
For me personally (and to start this conversation if necessary), I view distutils as the package in the stdlib that handles building extension modules in the stdlib for Python. That view means that if something doesn't service that explicit goal then it shouldn't go into distutils.
I think that's a reasonable place to draw the line for larger features and major refactorings, especially with pip injecting setuptools for "pip wheel" and "pip install", even if the setup.py itself only imports distutils.
However, in cases where distutils *does* provide a particular piece of functionality (e.g. the recent switch to emitting sdists as "tar.gz" by default, regardless of platform), then we should maintain a "distutils first" policy, make the change to distutils for future versions of Python, and then backport it to earlier versions via setuptools.
The grey area is cases like the one Sylvain is proposing, where it's a straightforward API addition to distutils with no backwards compatibility implications, and where we (as in distutils-sig/PyPA) want to encourage people to write build scripts a certain way (in this case, failing with a useful error message when particular compiler features aren't available, rather than confronting end users with a cryptic C/C++ compiler error).
For these cases, I think the value of including the change in distutils isn't so much a practical software development one, as it is an advertising opportunity: the What's New for 3.6 can not only mention the addition and recommendation for use, it can point people to setuptools as a version independent way of accessing the same capability. There are still a *lot* of people out there for whom the standard library *is* Python, and the feature releases are one of our best opportunities to reach those folks with notifications that the world has changed around them, and they may need to adapt their development practices accordingly.
The backwards compatibility situation for CPython feature releases itself also isn't as dire as it used to be - there are only a few tightly coupled distutils extensions (with setuptools and numpy.distutils being the most high profile ones), and it's rare these days for us to get through a beta period without finding and resolving any problems that come up (it's also worth remembering that the original major problem wasn't with a feature release, it was with a setuptools-breaking internal refactoring that landed in a maintenance release, so there was no beta period to find and fix compatibility problems).
Cheers, Nick.
On Aug 28, 2016, at 22:28, Nick Coghlan ncoghlan@gmail.com wrote: [...]
The grey area is cases like the one Sylvain is proposing, where it's a straightforward API addition to distutils with no backwards compatibility implications, and where we (as in distutils-sig/PyPA) want to encourage people to write build scripts a certain way (in this case, failing with a useful error message when particular compiler features aren't available, rather than confronting end users with a cryptic C/C++ compiler error).
For these cases, I think the value of including the change in distutils isn't so much a practical software development one, as it is an advertising opportunity: the What's New for 3.6 can not only mention the addition and recommendation for use, it can point people to setuptools as a version independent way of accessing the same capability. There are still a *lot* of people out there for whom the standard library *is* Python, and the feature releases are one of our best opportunities to reach those folks with notifications that the world has changed around them, and they may need to adapt their development practices accordingly.
I rebooted the discussion of Sylvain's http://bugs.python.org/issue26689 here because I think decisions about Distutils API changes rightly fall under the purview of the PyPA / Distutils SIG. With my 3.6 release manager hat on, I'd like to make sure there is agreement here before accepting this change for the release. So, my understanding is that an acceptable course of action, as outlined by Nick, would be to see the "has_flag" method supplied by Distutils for 3.6 *and* by setuptools for use in older releases. If so, I think the following things need to happen:
1. Someone from the PyPA (Nick?) should formally review and approve Sylvain's patch on the bug tracker.
2. Someone needs to open a setuptools issue and get agreement on the setuptools changes.
3. Someone needs to write the doc changes for 3.6, e.g. the What's New "advertising" and the changes to the Distutils documentation "Distributing Python Modules (Legacy version)" (Doc/distuils). Anywhere else?
Is that a fair summary?
Thanks for everyone's input on this! --Ned
-- Ned Deily nad@python.org -- []
participants (11)
-
Anna Ravenscroft -
Antoine Pitrou -
Brett Cannon -
Daniel Holth -
Ned Deily -
Nick Coghlan -
Paul Moore -
Ralf Gommers -
Robert Collins -
Steve Dower -
Sylvain Corlay