(FWIW, I think it makes *much* more sense for setuptools to fix this by simply forking all of distutils and never looking back. But since we don't
This was a comment by @zooba (Steve Dower): live in that world yet, it went into distutils.) And here is my response:
Since you mention it, I agree with that proposal. But currently we have core developers contributing to distutils and @jaraco contributing to setuptools. @jaraco is quite competent, but I doubt that he would be able to maintain an independent fork of distutils by himself.
In short, I think your proposal is a good one, but how can we allocate manpower?
(issue31595 on bugs.python.org) So what do others think of this? My sense of things is that people are open to the idea, but there isn't a plan to make it happen.
On Wed, Sep 27, 2017 at 3:30 PM, xoviat <xoviat@gmail.com> wrote:
This was a comment by @zooba (Steve Dower):
(FWIW, I think it makes *much* more sense for setuptools to fix this by simply forking all of distutils and never looking back. But since we don't live in that world yet, it went into distutils.)
And here is my response:
Since you mention it, I agree with that proposal. But currently we have core developers contributing to distutils and @jaraco contributing to setuptools. @jaraco is quite competent, but I doubt that he would be able to maintain an independent fork of distutils by himself.
In short, I think your proposal is a good one, but how can we allocate manpower?
(issue31595 on bugs.python.org)
So what do others think of this? My sense of things is that people are open to the idea, but there isn't a plan to make it happen.
My $2c: I'd only be a very occasional contributor, but it makes a lot of sense from the point of view of packaging/distributing changes to distutils. Also setuptools is a lot better maintained than distutils, and using the bug tracker is a much better experience. So many reasons to do it, and I'd certainly be more likely to report bugs and/or fix them. Ralf
On 27 September 2017 at 12:30, xoviat <xoviat@gmail.com> wrote:
In short, I think your proposal is a good one, but how can we allocate manpower?
(issue31595 on bugs.python.org)
So what do others think of this? My sense of things is that people are open to the idea, but there isn't a plan to make it happen.
As long as Jason is amenable to the idea on the setuptools side, and pip's trick of injecting setuptools into a process to change the behaviour of distutils imports continues to work, then this seems like a reasonable idea to me. There will still be folks that want to patch distutils itself, but I don't see that as being substantially different from folks still patching the standard library Tcl/Tk bindings, even though there are a number of popular non-stdlib GUI toolkits. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sep 27, 2017, at 5:37 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
pip's trick of injecting setuptools into a process to change the behaviour of distutils imports continues to work
I suspect this might be difficult, but I haven’t actually tried it. Our trick (and really setup tools itself relies on this too) is that setuptools monkey patches distutils at runtime to make distutils into setuptools. It only has to patch a few bits and pieces right now, but I suspect that if this happened it’d have to monkey patch the entire namespace. It would probably also break people who mix and match imports from setuptools and distutils. Of course, maybe it won’t! Won’t know until we try it I suppose.
On 27 September 2017 at 23:35, Donald Stufft <donald@stufft.io> wrote:
I suspect this might be difficult, but I haven’t actually tried it. Our trick (and really setup tools itself relies on this too) is that setuptools monkey patches distutils at runtime to make distutils into setuptools. It only has to patch a few bits and pieces right now, but I suspect that if this happened it’d have to monkey patch the entire namespace. It would probably also break people who mix and match imports from setuptools and distutils.
Right, my assumption was that pip would need to ensure that setuptools took over the distutils namespace before any of the code in setup.py ran, so that any monkeypatches and state modifications in other code would all take place in the setuptools provided copy, and the standard library's copy would never even get loaded. However, I don't think that's a new requirement: I believe you already need to do that, since the first line might be "from distutils import setup". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
So since PEP 517 is going into setuptools anyway, we can sys.path.insert(0, NEW_DISTUTILS_LOCATION) and then import distutils in the interface before setup.py is loaded. Seems simple enough. 2017-09-27 11:27 GMT-05:00 Nick Coghlan <ncoghlan@gmail.com>:
On 27 September 2017 at 23:35, Donald Stufft <donald@stufft.io> wrote:
I suspect this might be difficult, but I haven’t actually tried it. Our trick (and really setup tools itself relies on this too) is that setuptools monkey patches distutils at runtime to make distutils into setuptools. It only has to patch a few bits and pieces right now, but I suspect that if this happened it’d have to monkey patch the entire namespace. It would probably also break people who mix and match imports from setuptools and distutils.
Right, my assumption was that pip would need to ensure that setuptools took over the distutils namespace before any of the code in setup.py ran, so that any monkeypatches and state modifications in other code would all take place in the setuptools provided copy, and the standard library's copy would never even get loaded.
However, I don't think that's a new requirement: I believe you already need to do that, since the first line might be "from distutils import setup".
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
/cc @jaraco 2017-09-27 11:34 GMT-05:00 xoviat <xoviat@gmail.com>:
So since PEP 517 is going into setuptools anyway, we can sys.path.insert(0, NEW_DISTUTILS_LOCATION) and then import distutils in the interface before setup.py is loaded. Seems simple enough.
2017-09-27 11:27 GMT-05:00 Nick Coghlan <ncoghlan@gmail.com>:
On 27 September 2017 at 23:35, Donald Stufft <donald@stufft.io> wrote:
I suspect this might be difficult, but I haven’t actually tried it. Our trick (and really setup tools itself relies on this too) is that setuptools monkey patches distutils at runtime to make distutils into setuptools. It only has to patch a few bits and pieces right now, but I suspect that if this happened it’d have to monkey patch the entire namespace. It would probably also break people who mix and match imports from setuptools and distutils.
Right, my assumption was that pip would need to ensure that setuptools took over the distutils namespace before any of the code in setup.py ran, so that any monkeypatches and state modifications in other code would all take place in the setuptools provided copy, and the standard library's copy would never even get loaded.
However, I don't think that's a new requirement: I believe you already need to do that, since the first line might be "from distutils import setup".
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 27.09.2017 11:37, Nick Coghlan wrote:
On 27 September 2017 at 12:30, xoviat <xoviat@gmail.com> wrote:
In short, I think your proposal is a good one, but how can we allocate manpower?
(issue31595 on bugs.python.org)
So what do others think of this? My sense of things is that people are open to the idea, but there isn't a plan to make it happen.
As long as Jason is amenable to the idea on the setuptools side, and pip's trick of injecting setuptools into a process to change the behaviour of distutils imports continues to work, then this seems like a reasonable idea to me.
There will still be folks that want to patch distutils itself, but I don't see that as being substantially different from folks still patching the standard library Tcl/Tk bindings, even though there are a number of popular non-stdlib GUI toolkits.
In this case, we should deprecate distutils in python3.7 and remove it in 3.8.
On 29 September 2017 at 07:42, Matthias Klose <doko@ubuntu.com> wrote:
On 27.09.2017 11:37, Nick Coghlan wrote:
On 27 September 2017 at 12:30, xoviat <xoviat@gmail.com> wrote:
In short, I think your proposal is a good one, but how can we allocate manpower?
(issue31595 on bugs.python.org)
So what do others think of this? My sense of things is that people are open to the idea, but there isn't a plan to make it happen.
As long as Jason is amenable to the idea on the setuptools side, and pip's trick of injecting setuptools into a process to change the behaviour of distutils imports continues to work, then this seems like a reasonable idea to me.
There will still be folks that want to patch distutils itself, but I don't see that as being substantially different from folks still patching the standard library Tcl/Tk bindings, even though there are a number of popular non-stdlib GUI toolkits.
In this case, we should deprecate distutils in python3.7 and remove it in 3.8.
Aye, that seems reasonable to me as well (especially if 3.7+ also offers an "ensuresetuptools" module), and, looking at the recent(-ish) commit history at https://github.com/python/cpython/commits/master/Lib/distutils, I'd suggest we ask Steve Dower if he'd be willing to be BDFL-Delegate for the related PEP. This is one of those changes where the longer term process improvement benefits are already reasonably clear to the folks involved in maintaining the software (i.e. we think it will provide a better end user experience overall if distutils switches to being updated and versioned based on the setuptools release cycle rather than the reference interpreter & standard library one), so the main thing the PEP process will need to ensure is that we're providing a sufficiently non-disruptive transition plan for getting from the current state to the more desirable state. Between Steve ("Don't break Windows"), you ("Don't break Debian (et al)"), me ("Don't break Fedora (et al)"), Donald ("Don't break pip"), and Jason ("Don't break setuptools"/"Don't lumber setuptools with an unreasonable ongoing maintenance burden"), along with everyone else on distutils-sig, python-ideas, and python-dev, I think we'll be able to make a reasonable assessment of what counts as "too disruptive". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Actually, you can't remove any module in the stdlib that exists in both Python 2.7 and 3.5 until after Python 2.7 is no longer supported: https://www.python.org/dev/peps/pep-0004/#for-modules-existing-in-both-pytho... . On Thu, 28 Sep 2017 at 23:31 Nick Coghlan <ncoghlan@gmail.com> wrote:
On 29 September 2017 at 07:42, Matthias Klose <doko@ubuntu.com> wrote:
On 27.09.2017 11:37, Nick Coghlan wrote:
On 27 September 2017 at 12:30, xoviat <xoviat@gmail.com> wrote:
In short, I think your proposal is a good one, but how can we allocate manpower?
(issue31595 on bugs.python.org)
So what do others think of this? My sense of things is that people are open to the idea, but there isn't a plan to make it happen.
As long as Jason is amenable to the idea on the setuptools side, and pip's trick of injecting setuptools into a process to change the behaviour of distutils imports continues to work, then this seems like a reasonable idea to me.
There will still be folks that want to patch distutils itself, but I don't see that as being substantially different from folks still patching the standard library Tcl/Tk bindings, even though there are a number of popular non-stdlib GUI toolkits.
In this case, we should deprecate distutils in python3.7 and remove it in 3.8.
Aye, that seems reasonable to me as well (especially if 3.7+ also offers an "ensuresetuptools" module), and, looking at the recent(-ish) commit history at https://github.com/python/cpython/commits/master/Lib/distutils, I'd suggest we ask Steve Dower if he'd be willing to be BDFL-Delegate for the related PEP.
This is one of those changes where the longer term process improvement benefits are already reasonably clear to the folks involved in maintaining the software (i.e. we think it will provide a better end user experience overall if distutils switches to being updated and versioned based on the setuptools release cycle rather than the reference interpreter & standard library one), so the main thing the PEP process will need to ensure is that we're providing a sufficiently non-disruptive transition plan for getting from the current state to the more desirable state.
Between Steve ("Don't break Windows"), you ("Don't break Debian (et al)"), me ("Don't break Fedora (et al)"), Donald ("Don't break pip"), and Jason ("Don't break setuptools"/"Don't lumber setuptools with an unreasonable ongoing maintenance burden"), along with everyone else on distutils-sig, python-ideas, and python-dev, I think we'll be able to make a reasonable assessment of what counts as "too disruptive".
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
I understand the motivation behind that PEP, but we're not really deprecating distutils, just moving it out into setuptools. The goal should be "not to break anyone's code." 2017-09-29 11:25 GMT-05:00 Brett Cannon <brett@python.org>:
Actually, you can't remove any module in the stdlib that exists in both Python 2.7 and 3.5 until after Python 2.7 is no longer supported: https://www.python.org/dev/peps/pep-0004/#for-modules- existing-in-both-python-2-7-and-python-3-5 .
On Thu, 28 Sep 2017 at 23:31 Nick Coghlan <ncoghlan@gmail.com> wrote:
On 29 September 2017 at 07:42, Matthias Klose <doko@ubuntu.com> wrote:
On 27.09.2017 11:37, Nick Coghlan wrote:
On 27 September 2017 at 12:30, xoviat <xoviat@gmail.com> wrote:
In short, I think your proposal is a good one, but how can we allocate manpower?
(issue31595 on bugs.python.org)
So what do others think of this? My sense of things is that people are open to the idea, but there isn't a plan to make it happen.
As long as Jason is amenable to the idea on the setuptools side, and pip's trick of injecting setuptools into a process to change the behaviour of distutils imports continues to work, then this seems like a reasonable idea to me.
There will still be folks that want to patch distutils itself, but I don't see that as being substantially different from folks still patching the standard library Tcl/Tk bindings, even though there are a number of popular non-stdlib GUI toolkits.
In this case, we should deprecate distutils in python3.7 and remove it in 3.8.
Aye, that seems reasonable to me as well (especially if 3.7+ also offers an "ensuresetuptools" module), and, looking at the recent(-ish) commit history at https://github.com/python/cpython/commits/master/Lib/distutils, I'd suggest we ask Steve Dower if he'd be willing to be BDFL-Delegate for the related PEP.
This is one of those changes where the longer term process improvement benefits are already reasonably clear to the folks involved in maintaining the software (i.e. we think it will provide a better end user experience overall if distutils switches to being updated and versioned based on the setuptools release cycle rather than the reference interpreter & standard library one), so the main thing the PEP process will need to ensure is that we're providing a sufficiently non-disruptive transition plan for getting from the current state to the more desirable state.
Between Steve ("Don't break Windows"), you ("Don't break Debian (et al)"), me ("Don't break Fedora (et al)"), Donald ("Don't break pip"), and Jason ("Don't break setuptools"/"Don't lumber setuptools with an unreasonable ongoing maintenance burden"), along with everyone else on distutils-sig, python-ideas, and python-dev, I think we'll be able to make a reasonable assessment of what counts as "too disruptive".
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Sep 29, 2017, at 2:31 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
This is one of those changes where the longer term process improvement benefits are already reasonably clear to the folks involved in maintaining the software (i.e. we think it will provide a better end user experience overall if distutils switches to being updated and versioned based on the setuptools release cycle rather than the reference interpreter & standard library one), so the main thing the PEP process will need to ensure is that we're providing a sufficiently non-disruptive transition plan for getting from the current state to the more desirable state.
I’m not sure that it’s going to be helpful to take distutils out of the standard library. It has many problems, and being in the standard library is one of them sure, but the more insidious problems are inherent in it’s entire design which we can’t really fix without breaking all of the things. The main blocker to improvement is primarily just that after two decades of existence tons of random setup.py scripts out there have reached in and messed with sometimes even super internal parts of distutils. There are bits of distutils that are super useful (Steve indicated the compiler support as one of them), and I think a better path forward is to do like we’ve done with the packaging library. Make a compiler package that can handle those bits and recommend that projects that want to handle compilation just reuse them (of course not mandate it!). Then we can focus first on creating a good API in that package without inheriting the problems of distutils, and then we can maybe work on bundling that inside of distutils and turn the distutils API into a shim over that (or just leaving it alone). This leaves the question of what happens to setuptools since it is intimately tied to distutils. I don’t really have a good answer for that. Probably it’s best to keep setuptools moving along as it’s doing now without making major architectural changes to it. It inherited a lot of the problems of distutils where people reach in and muck around with it’s internals so it’s easy to break things if you make too many changes. Therefore we might be better to just mostly leave distutils/setuptools progressing as they are today and focus on a brighter future in the greenfield landscape of PEP 517.
On 29Sep2017 1306, Donald Stufft wrote:
There are bits of distutils that are super useful (Steve indicated the compiler support as one of them), and I think a better path forward is to do like we’ve done with the packaging library. Make a compiler package that can handle those bits and recommend that projects that want to handle compilation just reuse them (of course not mandate it!). Then we can focus first on creating a good API in that package without inheriting the problems of distutils, and then we can maybe work on bundling that inside of distutils and turn the distutils API into a shim over that (or just leaving it alone).
I'm happy enough with this approach, my only problem with it is that I don't want to be maintaining two versions (the new one that we want people to change to, and the old one so that people keep working with new versions of Python without having to change to the new one). If we don't make a very clear statement that "distutils is no longer being maintained, it probably won't build valid extensions on 3.[7/8/whatever], and we mean it this time" then it doesn't count. I insist on at least removing the distutils tests from the test suite, since as long as those need to pass I can never stop maintaining the stdlib version. And that puts this in the realm of python-dev, which means a PEP to formally deprecate distutils (that I'm happy to co-author) and recommend an alternative, as well as the alternative path needed to build the core extension modules on non-Windows platforms. Whatever breaks setuptools's dependency on distutils helps drastically with this, as then we can recommend setuptools as the alternative path. Perhaps we can extract distutils into an installable package that will shadow the stdlib one, and setuptools can continue monkey patching while also taking a build/install-time dependency on it. I honestly don't know what the best approach here is, which is why I raise the question. I'm fairly convinced that the best approach is *not* to have a whole lot of PEP 517 build tools that depend on distutils for core functionality. Cheers, Steve
On Sep 29, 2017, at 5:16 PM, Steve Dower <steve.dower@python.org> wrote:
I'm happy enough with this approach, my only problem with it is that I don't want to be maintaining two versions (the new one that we want people to change to, and the old one so that people keep working with new versions of Python without having to change to the new one).
If we don't make a very clear statement that "distutils is no longer being maintained, it probably won't build valid extensions on 3.[7/8/whatever], and we mean it this time" then it doesn't count. I insist on at least removing the distutils tests from the test suite, since as long as those need to pass I can never stop maintaining the stdlib version. And that puts this in the realm of python-dev, which means a PEP to formally deprecate distutils (that I'm happy to co-author) and recommend an alternative, as well as the alternative path needed to build the core extension modules on non-Windows platforms.
As long as CPython itself depends on distutils I don’t think we can break or remove distutils, but we can move make it a shim over something that is better architecturally and easier to maintain. The tests should still pass in this case because they’d essentially be testing that our shim is functioning correctly. I also don’t think we can spin distutils out of the stdlib as long as CPython depends on it. I see two paths forward: 1) We come up with something better *in the standard library*, implement it, move CPython to using that in the build process, and then deprecate/remove distutils from the stdlib and have it maintained outside. 2) We come up with something better outside of the standard library, implement it, vendor it into distutils/_whatevercoollib, and turn the relevant distutils APIs into shims over the new API. Of the two, I think that (2) is going to be the least painful for users.
I don't think writing shims for distutils is as easy as you make it sound. In fact, I'd venture to say that it's an intractable problem because of the difficulty of knowing the number of distutils hacks that are in the wild. 2017-09-29 22:15 GMT-05:00 Donald Stufft <donald@stufft.io>:
On Sep 29, 2017, at 5:16 PM, Steve Dower <steve.dower@python.org> wrote:
I'm happy enough with this approach, my only problem with it is that I don't want to be maintaining two versions (the new one that we want people to change to, and the old one so that people keep working with new versions of Python without having to change to the new one).
If we don't make a very clear statement that "distutils is no longer being maintained, it probably won't build valid extensions on 3.[7/8/whatever], and we mean it this time" then it doesn't count. I insist on at least removing the distutils tests from the test suite, since as long as those need to pass I can never stop maintaining the stdlib version. And that puts this in the realm of python-dev, which means a PEP to formally deprecate distutils (that I'm happy to co-author) and recommend an alternative, as well as the alternative path needed to build the core extension modules on non-Windows platforms.
As long as CPython itself depends on distutils I don’t think we can break or remove distutils, but we can move make it a shim over something that is better architecturally and easier to maintain. The tests should still pass in this case because they’d essentially be testing that our shim is functioning correctly.
I also don’t think we can spin distutils out of the stdlib as long as CPython depends on it.
I see two paths forward:
1) We come up with something better *in the standard library*, implement it, move CPython to using that in the build process, and then deprecate/remove distutils from the stdlib and have it maintained outside.
2) We come up with something better outside of the standard library, implement it, vendor it into distutils/_whatevercoollib, and turn the relevant distutils APIs into shims over the new API.
Of the two, I think that (2) is going to be the least painful for users.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Sep 29, 2017, at 11:21 PM, xoviat <xoviat@gmail.com> wrote:
I don't think writing shims for distutils is as easy as you make it sound. In fact, I'd venture to say that it's an intractable problem because of the difficulty of knowing the number of distutils hacks that are in the wild.
For the compiler code? It’s probably pretty tractable, which is I think the main thing that needs maintenance.
I don't know whether this will convince anyone otherwise, but at least from my perspective, a build-time dependency is different than a run-time dependency. Assuming that other parts of CPython don't have a run-time dependency on distutils, I don't think CPython needs to bundle all of its build-time dependencies. That principle doesn't really apply to other Python programs nor most other programs in general. AFAIK, CPython already has a build-time dependency on another, external, Python, so it wouldn't be too much to require the external Python to have setuptools installed with something like pyproject.toml (other programming languages usually bootstrap themselves with previous versions of the language along with some associated build tools). 2017-09-29 22:43 GMT-05:00 Donald Stufft <donald@stufft.io>:
On Sep 29, 2017, at 11:21 PM, xoviat <xoviat@gmail.com> wrote:
I don't think writing shims for distutils is as easy as you make it sound. In fact, I'd venture to say that it's an intractable problem because of the difficulty of knowing the number of distutils hacks that are in the wild.
For the compiler code? It’s probably pretty tractable, which is I think the main thing that needs maintenance.
On Sep 30, 2017, at 3:52 PM, xoviat <xoviat@gmail.com> wrote:
I don't think CPython needs to bundle all of its build-time dependencies. That principle doesn't really apply to other Python programs nor most other programs in general. AFAIK, CPython already has a build-time dependency on another, external, Python, so it wouldn't be too much to require the external Python to have setuptools installed with something like pyproject.toml (other programming languages usually bootstrap themselves with previous versions of the language along with some associated build tools).
As far as I can tell, CPython does *not* have a build time dependency on having Python available. I just spun up a bare alpine linux container and compiled CPython `master` branch on it. As far as I can tell the only Python that exists in this container is the one I just compiled. That means that in order for CPython to depend on distutils to build as you indicate, it would also need to start depending on an existing version of Python being available. I don’t think that’s a great idea. I think Python should not depend on Python to build.
I have personally not built Python myself (though I've built many an extension), but what I can say is that I got the idea from Larry Hastings. According to him (this if for the Gilectomy fork): "Second, as you hack on the Gilectomy you may break your "python" executable rather badly. This is of course expected. However, the python Makefile itself depends on having a working local python interpreter, so when you break that you often break your build too." 2017-09-30 19:59 GMT-05:00 Donald Stufft <donald@stufft.io>:
On Sep 30, 2017, at 3:52 PM, xoviat <xoviat@gmail.com> wrote:
I don't think CPython needs to bundle all of its build-time dependencies. That principle doesn't really apply to other Python programs nor most other programs in general. AFAIK, CPython already has a build-time dependency on another, external, Python, so it wouldn't be too much to require the external Python to have setuptools installed with something like pyproject.toml (other programming languages usually bootstrap themselves with previous versions of the language along with some associated build tools).
As far as I can tell, CPython does *not* have a build time dependency on having Python available. I just spun up a bare alpine linux container and compiled CPython `master` branch on it. As far as I can tell the only Python that exists in this container is the one I just compiled.
That means that in order for CPython to depend on distutils to build as you indicate, it would also need to start depending on an existing version of Python being available. I don’t think that’s a great idea. I think Python should not depend on Python to build.
It would be nice to know whether this information is correct, or whether I hold an invalid belief. 2017-09-30 20:09 GMT-05:00 xoviat <xoviat@gmail.com>:
I have personally not built Python myself (though I've built many an extension), but what I can say is that I got the idea from Larry Hastings. According to him (this if for the Gilectomy fork):
"Second, as you hack on the Gilectomy you may break your "python" executable rather badly. This is of course expected. However, the python Makefile itself depends on having a working local python interpreter, so when you break that you often break your build too."
2017-09-30 19:59 GMT-05:00 Donald Stufft <donald@stufft.io>:
On Sep 30, 2017, at 3:52 PM, xoviat <xoviat@gmail.com> wrote:
I don't think CPython needs to bundle all of its build-time dependencies. That principle doesn't really apply to other Python programs nor most other programs in general. AFAIK, CPython already has a build-time dependency on another, external, Python, so it wouldn't be too much to require the external Python to have setuptools installed with something like pyproject.toml (other programming languages usually bootstrap themselves with previous versions of the language along with some associated build tools).
As far as I can tell, CPython does *not* have a build time dependency on having Python available. I just spun up a bare alpine linux container and compiled CPython `master` branch on it. As far as I can tell the only Python that exists in this container is the one I just compiled.
That means that in order for CPython to depend on distutils to build as you indicate, it would also need to start depending on an existing version of Python being available. I don’t think that’s a great idea. I think Python should not depend on Python to build.
I think that the CPython builds a python executable, then uses that built executable to finish the installation.
On Sep 30, 2017, at 9:11 PM, xoviat <xoviat@gmail.com> wrote:
It would be nice to know whether this information is correct, or whether I hold an invalid belief.
2017-09-30 20:09 GMT-05:00 xoviat <xoviat@gmail.com <mailto:xoviat@gmail.com>>: I have personally not built Python myself (though I've built many an extension), but what I can say is that I got the idea from Larry Hastings. According to him (this if for the Gilectomy fork):
"Second, as you hack on the Gilectomy you may break your "python" executable rather badly. This is of course expected. However, the python Makefile itself depends on having a working local python interpreter, so when you break that you often break your build too."
2017-09-30 19:59 GMT-05:00 Donald Stufft <donald@stufft.io <mailto:donald@stufft.io>>:
On Sep 30, 2017, at 3:52 PM, xoviat <xoviat@gmail.com <mailto:xoviat@gmail.com>> wrote:
I don't think CPython needs to bundle all of its build-time dependencies. That principle doesn't really apply to other Python programs nor most other programs in general. AFAIK, CPython already has a build-time dependency on another, external, Python, so it wouldn't be too much to require the external Python to have setuptools installed with something like pyproject.toml (other programming languages usually bootstrap themselves with previous versions of the language along with some associated build tools).
As far as I can tell, CPython does *not* have a build time dependency on having Python available. I just spun up a bare alpine linux container and compiled CPython `master` branch on it. As far as I can tell the only Python that exists in this container is the one I just compiled.
That means that in order for CPython to depend on distutils to build as you indicate, it would also need to start depending on an existing version of Python being available. I don’t think that’s a great idea. I think Python should not depend on Python to build.
It must, or it couldn’t have a build time dependency on distutils :) Top-posted from my Windows phone From: Donald Stufft Sent: Saturday, September 30, 2017 18:15 To: xoviat Cc: Steve Dower; DistUtils mailing list Subject: Re: [Distutils] Extracting distutils into setuptools I think that the CPython builds a python executable, then uses that built executable to finish the installation. On Sep 30, 2017, at 9:11 PM, xoviat <xoviat@gmail.com> wrote: It would be nice to know whether this information is correct, or whether I hold an invalid belief. 2017-09-30 20:09 GMT-05:00 xoviat <xoviat@gmail.com>: I have personally not built Python myself (though I've built many an extension), but what I can say is that I got the idea from Larry Hastings. According to him (this if for the Gilectomy fork): "Second, as you hack on the Gilectomy you may break your "python" executable rather badly. This is of course expected. However, the python Makefile itself depends on having a working local python interpreter, so when you break that you often break your build too." 2017-09-30 19:59 GMT-05:00 Donald Stufft <donald@stufft.io>: On Sep 30, 2017, at 3:52 PM, xoviat <xoviat@gmail.com> wrote: I don't think CPython needs to bundle all of its build-time dependencies. That principle doesn't really apply to other Python programs nor most other programs in general. AFAIK, CPython already has a build-time dependency on another, external, Python, so it wouldn't be too much to require the external Python to have setuptools installed with something like pyproject.toml (other programming languages usually bootstrap themselves with previous versions of the language along with some associated build tools). As far as I can tell, CPython does *not* have a build time dependency on having Python available. I just spun up a bare alpine linux container and compiled CPython `master` branch on it. As far as I can tell the only Python that exists in this container is the one I just compiled. That means that in order for CPython to depend on distutils to build as you indicate, it would also need to start depending on an existing version of Python being available. I don’t think that’s a great idea. I think Python should not depend on Python to build.
After thinking again about that possibilities that we've discussed here, I realized that a previously proposed alternative would eliminate external build-time dependencies and allow us to merge setuptools with distutils: an "ensuresetuptools" module. This was proposed by @zooba, but basically the idea would be to bundle a wheel of setuptools (setuptools is py2.py3.none-any) that Python could install without requiring network access or other modules. If distutils is required during the build process, then this idea should conform to all of the requirements proposed here and take distutils off of the CPython release schedule. Although it may seem more complicated at first, once implemented, maintenance from the CPython side would be minimal (a bot could update the wheel, although I'm not sure whether this is done with ensurepip), there is precedent for this (with ensurepip), end-users could continue to use distutils without any modifications to their scripts (ensuresetuptools could be run during the installation process, and even if it isn't, then a single command line could install distutils/setuptools), and it would allow simplified setuptools maintenance (no monkeypatching). 2017-09-30 20:14 GMT-05:00 Donald Stufft <donald@stufft.io>:
I think that the CPython builds a python executable, then uses that built executable to finish the installation.
On Sep 30, 2017, at 9:11 PM, xoviat <xoviat@gmail.com> wrote:
It would be nice to know whether this information is correct, or whether I hold an invalid belief.
2017-09-30 20:09 GMT-05:00 xoviat <xoviat@gmail.com>:
I have personally not built Python myself (though I've built many an extension), but what I can say is that I got the idea from Larry Hastings. According to him (this if for the Gilectomy fork):
"Second, as you hack on the Gilectomy you may break your "python" executable rather badly. This is of course expected. However, the python Makefile itself depends on having a working local python interpreter, so when you break that you often break your build too."
2017-09-30 19:59 GMT-05:00 Donald Stufft <donald@stufft.io>:
On Sep 30, 2017, at 3:52 PM, xoviat <xoviat@gmail.com> wrote:
I don't think CPython needs to bundle all of its build-time dependencies. That principle doesn't really apply to other Python programs nor most other programs in general. AFAIK, CPython already has a build-time dependency on another, external, Python, so it wouldn't be too much to require the external Python to have setuptools installed with something like pyproject.toml (other programming languages usually bootstrap themselves with previous versions of the language along with some associated build tools).
As far as I can tell, CPython does *not* have a build time dependency on having Python available. I just spun up a bare alpine linux container and compiled CPython `master` branch on it. As far as I can tell the only Python that exists in this container is the one I just compiled.
That means that in order for CPython to depend on distutils to build as you indicate, it would also need to start depending on an existing version of Python being available. I don’t think that’s a great idea. I think Python should not depend on Python to build.
On Oct 1, 2017, at 1:53 PM, xoviat <xoviat@gmail.com> wrote:
After thinking again about that possibilities that we've discussed here, I realized that a previously proposed alternative would eliminate external build-time dependencies and allow us to merge setuptools with distutils: an "ensuresetuptools" module. This was proposed by @zooba, but basically the idea would be to bundle a wheel of setuptools (setuptools is py2.py3.none-any) that Python could install without requiring network access or other modules. If distutils is required during the build process, then this idea should conform to all of the requirements proposed here and take distutils off of the CPython release schedule.
This isn’t as easy as ensurepip, because ensurepip can wait until the end of the build process when the entire Python installation has been built. The same isn’t true for a hypothetical ensuresetuptools module. This is because we end up in a circular dependency, if installing a wheel requires a C extension (like say zlib) then we can’t install that wheel prior to building zlib, but if we need to install that wheel to build zlib then we end up stuck. I’m not sure what all c-extensions are used in the process of installing a wheel— certainly zlib is but we could maybe build a special wheel that only uses the STORED algorithm and doesn’t do compression (does the zipfile module work with zlib doesn’t exist?). I’m going to guess there are others though and no idea if they are able to be avoided. I still think a better idea if we want to go down that route is to modify the CPython build process to not depend on distutils at all.
On 2 October 2017 at 04:55, Donald Stufft <donald@stufft.io> wrote:
This isn’t as easy as ensurepip, because ensurepip can wait until the end of the build process when the entire Python installation has been built. The same isn’t true for a hypothetical ensuresetuptools module. This is because we end up in a circular dependency, if installing a wheel requires a C extension (like say zlib) then we can’t install that wheel prior to building zlib, but if we need to install that wheel to build zlib then we end up stuck.
I’m not sure what all c-extensions are used in the process of installing a wheel— certainly zlib is but we could maybe build a special wheel that only uses the STORED algorithm and doesn’t do compression (does the zipfile module work with zlib doesn’t exist?). I’m going to guess there are others though and no idea if they are able to be avoided.
I still think a better idea if we want to go down that route is to modify the CPython build process to not depend on distutils at all.
The main reason for using distutils is to reduce the need to define different build instructions for different platforms for things like curses, sqlite, cdecimal, dbm, Tcl/Tk, etc. (And even when we do need to define separate instructions for Windows, we still get to avoid running those builds through autotools) That said, we do have other extension modules that *are* handled by the platform specific build instructions, and it should be relatively straightforward to migrate anything that `ensuresetuptools` needed down to that layer (and it's likely only `zlib` that falls into that category). Cheers, Nick. P.S. I'm honestly not sure how well CPython in general works without zlib: while it's nominally optional in the build process, and we have a "test.support.requires_zlib" decorator, I don't think we run any buildbots without it, and we statically link it on Windows and Mac OS X. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 1 October 2017 at 10:59, Donald Stufft <donald@stufft.io> wrote:
On Sep 30, 2017, at 3:52 PM, xoviat <xoviat@gmail.com> wrote:
I don't think CPython needs to bundle all of its build-time dependencies. That principle doesn't really apply to other Python programs nor most other programs in general. AFAIK, CPython already has a build-time dependency on another, external, Python, so it wouldn't be too much to require the external Python to have setuptools installed with something like pyproject.toml (other programming languages usually bootstrap themselves with previous versions of the language along with some associated build tools).
As far as I can tell, CPython does *not* have a build time dependency on having Python available. I just spun up a bare alpine linux container and compiled CPython `master` branch on it. As far as I can tell the only Python that exists in this container is the one I just compiled.
That means that in order for CPython to depend on distutils to build as you indicate, it would also need to start depending on an existing version of Python being available. I don’t think that’s a great idea. I think Python should not depend on Python to build.
CPython's self-bootstrapping is a bit complicated :)
From a regular checkout, the way it works is:
1. We build CPython itself, including all frozen and builtin modules, since the source code for the frozen modules is checked in, as is the code generated by Argument Clinic 2. We use distutils & that just built CPython to build a bunch of optional extension modules For working on the compiler itself, we also have an extra step where we build CPython with no import system, use that to refreeze importlib with the modified compiler, then rebuild CPython with the recompiled import system. Breaking the code to run the code generators can thus break your build, although that should be less common now that we require execution of the code generators to be requested explicitly (and rely on CI to check when they haven't been run when they should). However, there's *nothing* in that arrangement which says we can't use a virtual environment to do the optional extension module builds, and plenty of good arguments to say that we *should* be using a virtual environment for those. And if we move them into a virtual environment, then *those builds* will still have access to distutils, regardless of whether we're picking it up from Lib/distutils (which is what we currently do), or from an installed setuptools (which is what I'd propose we switch to for 3.8+). As far as PEP 4 goes, I think the constraint that places on us is that *after installation of CPython*, then "import distutils" has to continue working, just as it does today. I *don't* think it places any requirement on us to ensure that "Lib/distutils" is a thing that exists in the CPython source repo - I'd consider it entirely acceptable for us to install it into site-packages with `pip` (PEP 453 style), and have it be missing by default from things like the embeddable Windows distribution. As far as the intended pay-off goes, this is entirely aimed at reduce the level of work for the current setuptools maintainers, as they have a complex bug triage process that requires them to do something like this: 1. Figure out if something is a distutils bug or a setuptools bug 2. If it's a distutils bug, figure out if setuptools can work around it 3. Regardless of whether setuptools can work around it or not, file a bug against distutils on b.p.o 4. Decide which branches to fix the distutils bug on 5. If setuptools can't work around it, tell people they'll have to upgrade CPython to fix their problem setuptools forking distutils wholesale simplifies that whole triage & process to: 1. It doesn't matter if the problem is a distutils bug or a setuptools bug, fix it in the setuptools code base 2. Ship a new release of setuptools to make the fix available to users However, if setuptools *does* fork distutils wholesale, then the already low level of contributions to distutils maintenance are likely to drop to essentially zero, at which point the question becomes "How can we switch *CPython itself* to using and providing the independently upgradable setuptools fork instead of the Lib/distutils version?" And there, PEP 453 has already established a relevant precedent: we can bundle setuptools as a wheel file, with a helper module to request installation of that wheel file's contents into the current environment. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Distutils has many problems as you said, but by direct implication, if setuptools is "intimately tied" to distutils, then setuptools has these problems as well (and I think that it does).
I don’t really have a good answer for that. Probably it’s best to keep setuptools moving along as it’s doing now without making major architectural changes to it.
The answer (I think) is clear. Setuptools should take over the distutils namespace, even on older Python versions, and even when there is already a distutils in the standard library. If setuptools is imported first, then it can rewrite the import paths such that the setuptools version of distutils is used. This would extract and consolidate what has become a de-facto single project (distutils + setuptools) into one repository, reducing duplication of effort. 2017-09-29 15:06 GMT-05:00 Donald Stufft <donald@stufft.io>:
On Sep 29, 2017, at 2:31 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
This is one of those changes where the longer term process improvement benefits are already reasonably clear to the folks involved in maintaining the software (i.e. we think it will provide a better end user experience overall if distutils switches to being updated and versioned based on the setuptools release cycle rather than the reference interpreter & standard library one), so the main thing the PEP process will need to ensure is that we're providing a sufficiently non-disruptive transition plan for getting from the current state to the more desirable state.
I’m not sure that it’s going to be helpful to take distutils out of the standard library. It has many problems, and being in the standard library is one of them sure, but the more insidious problems are inherent in it’s entire design which we can’t really fix without breaking all of the things. The main blocker to improvement is primarily just that after two decades of existence tons of random setup.py scripts out there have reached in and messed with sometimes even super internal parts of distutils.
There are bits of distutils that are super useful (Steve indicated the compiler support as one of them), and I think a better path forward is to do like we’ve done with the packaging library. Make a compiler package that can handle those bits and recommend that projects that want to handle compilation just reuse them (of course not mandate it!). Then we can focus first on creating a good API in that package without inheriting the problems of distutils, and then we can maybe work on bundling that inside of distutils and turn the distutils API into a shim over that (or just leaving it alone).
This leaves the question of what happens to setuptools since it is intimately tied to distutils. I don’t really have a good answer for that. Probably it’s best to keep setuptools moving along as it’s doing now without making major architectural changes to it. It inherited a lot of the problems of distutils where people reach in and muck around with it’s internals so it’s easy to break things if you make too many changes.
Therefore we might be better to just mostly leave distutils/setuptools progressing as they are today and focus on a brighter future in the greenfield landscape of PEP 517.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 26Sep2017 1930, xoviat wrote:
This was a comment by @zooba (Steve Dower):
(FWIW, I think it makes *much* more sense for setuptools to fix this by simply forking all of distutils and never looking back. But since we don't live in that world yet, it went into distutils.)
And here is my response:
Since you mention it, I agree with that proposal. But currently we have core developers contributing to distutils and @jaraco contributing to setuptools. @jaraco is quite competent, but I doubt that he would be able to maintain an independent fork of distutils by himself.
In short, I think your proposal is a good one, but how can we allocate manpower?
(issue31595 on bugs.python.org <http://bugs.python.org>)
So what do others think of this? My sense of things is that people are open to the idea, but there isn't a plan to make it happen.
Just to put it out there, I know this is asking a lot of Jason and the setuptools maintainers. I don't make the proposal lightly. But especially in the world of PEP 517 I would rather not have an array of new libraries depending on distutils for their ability to compile extension modules. As long as there is "core" functionality to do this, we won't ever get any innovation out there. For an example of innovation, I've already published a library that can find VS 2015 and VS 2017 installs on Windows, generate the correct project (make) files and build an extension.[1] It's far more reliable and customisable than distutils - and I've put barely any effort into it! - but as long as people have to choose between a dependency and a core library nobody will ever get to take advantage of it. (And, as occurred, I'll end up porting pieces into core CPython so that we don't break the world by *really* deprecating distutils.) setuptools is totally welcome in my book to simply copy the compiler infrastructure we already have from core and never look back. It really does need to be maintained separately from CPython, especially on Windows where we continue to get innovation in the targeted tools. I know it's a big ask, and it's one that I can't personally commit real time to (though I obviously will as much as possible), but I do think it is necessary for our ecosystem to not be tied to CPython release cycles. Cheers, Steve [1]: The library is at https://pypi.org/project/pyfindvs/ and the docs do not refer to the compilation part at all. Look at https://github.com/zooba/pyfindvs/tree/master/pyfindvs/msbuildcompiler to see it.
On Sep 27, 2017, at 13:31, Steve Dower <steve.dower@python.org> wrote:
setuptools is totally welcome in my book to simply copy the compiler infrastructure we already have from core and never look back. It really does need to be maintained separately from CPython, especially on Windows where we continue to get innovation in the targeted tools. I know it's a big ask, and it's one that I can't personally commit real time to (though I obviously will as much as possible), but I do think it is necessary for our ecosystem to not be tied to CPython release cycles.
Whatever is done, keep in mind that currently distutils is required to build Python itself, e.g. the standard library. And that at least one important project, numpy, already subclasses distutils. -- Ned Deily nad@python.org -- []
No. Setuptools is what projects without a build_backend in pyproject.toml get. Not distutils. We should make it clear now that the distutils namespace belongs to setuptools except for when building cpython. On Sep 27, 2017 2:33 PM, "Ned Deily" <nad@python.org> wrote:
On Sep 27, 2017, at 13:31, Steve Dower <steve.dower@python.org> wrote:
setuptools is totally welcome in my book to simply copy the compiler infrastructure we already have from core and never look back. It really does need to be maintained separately from CPython, especially on Windows where we continue to get innovation in the targeted tools. I know it's a big ask, and it's one that I can't personally commit real time to (though I obviously will as much as possible), but I do think it is necessary for our ecosystem to not be tied to CPython release cycles.
Whatever is done, keep in mind that currently distutils is required to build Python itself, e.g. the standard library. And that at least one important project, numpy, already subclasses distutils.
-- Ned Deily nad@python.org -- []
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
That's actually an interesting idea though: for Python 3.7 distutils -> _distutils (and then setuptools is required for building). For/against? 2017-09-27 14:46 GMT-05:00 xoviat <xoviat@gmail.com>:
No. Setuptools is what projects without a build_backend in pyproject.toml get. Not distutils. We should make it clear now that the distutils namespace belongs to setuptools except for when building cpython.
On Sep 27, 2017 2:33 PM, "Ned Deily" <nad@python.org> wrote:
On Sep 27, 2017, at 13:31, Steve Dower <steve.dower@python.org> wrote:
setuptools is totally welcome in my book to simply copy the compiler infrastructure we already have from core and never look back. It really does need to be maintained separately from CPython, especially on Windows where we continue to get innovation in the targeted tools. I know it's a big ask, and it's one that I can't personally commit real time to (though I obviously will as much as possible), but I do think it is necessary for our ecosystem to not be tied to CPython release cycles.
Whatever is done, keep in mind that currently distutils is required to build Python itself, e.g. the standard library. And that at least one important project, numpy, already subclasses distutils.
-- Ned Deily nad@python.org -- []
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 28 September 2017 at 06:00, xoviat <xoviat@gmail.com> wrote:
That's actually an interesting idea though: for Python 3.7 distutils -> _distutils (and then setuptools is required for building). For/against?
distutils works fine for its original purpose (building components for the system Python in Linux distros), so we still need to avoid breaking that. setuptools is only essential if you want full support for modern *Python* level packaging features (PEP 376 install metadata, venv compatibility, wheel files, etc), and a lot of Linux system components simply don't worry about those things, and rely on their system level equivalents instead (e.g. the RPM/deb databases, chroots and containers, RPM/deb files) However, what *could* be interesting is a proposal to move distutils to the "ensurepip" model, where rather than maintaining distutils directly as part of CPython, the CPython build process instead runs setuptools/distutils from a bundled wheel file. Doing that would entail having setuptools actually start installing a copy of distutils into site-packages: older CPython releases would ignore it by default (since the stdlib version would shadow it), while 3.7+ would offer either "python3 -m ensuredistutils" or "python3 -m ensuresetuptools" (bikeshed to be painted via the PEP process) to install the setuptools-provided version. I'm not claiming actually doing that would be particularly easy - I just think it's the most viable path to get us away from the current version coupling between the build infrastructure in distutils and the runtime support infrastructure in the rest of the standard library, and to avoid maintaining two distinct copies of distutils indefinitely (one in the stdlib, one in setuptools). That approach wouldn't even entail any *new* bundling at the CPython level, as while it's currently formally an implementation detail (pending potential removal in a post-PEP-517 world), setuptools is already bundled as part of the support infrastructure for ensurepip: https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Enscons uses parts of distutils to get compiler flags and so on but does not use Extension() to do the actual compiling. There might be a cleaner way to do it that I was not able to find. There could be a cleaner separation between parts of distutils related to how Python itself was compiled and the part that does the actual compiling. The sysconfig module attempts to do at least part of this. distutils also knows about some compiler tricks like generating symbol tables for Windows, that are not easy to re-use or invoke separately. Someone could catalog these tricks and techniques and refactor the useful ones into a reusable library. I don't have a great sense of how to do it. https://bitbucket.org/dholth/enscons/src/tip/enscons/cpyext.py?at=default&fileviewer=file-view-default On Wed, Sep 27, 2017 at 11:36 PM Nick Coghlan <ncoghlan@gmail.com> wrote:
On 28 September 2017 at 06:00, xoviat <xoviat@gmail.com> wrote:
That's actually an interesting idea though: for Python 3.7 distutils -> _distutils (and then setuptools is required for building). For/against?
distutils works fine for its original purpose (building components for the system Python in Linux distros), so we still need to avoid breaking that. setuptools is only essential if you want full support for modern *Python* level packaging features (PEP 376 install metadata, venv compatibility, wheel files, etc), and a lot of Linux system components simply don't worry about those things, and rely on their system level equivalents instead (e.g. the RPM/deb databases, chroots and containers, RPM/deb files)
However, what *could* be interesting is a proposal to move distutils to the "ensurepip" model, where rather than maintaining distutils directly as part of CPython, the CPython build process instead runs setuptools/distutils from a bundled wheel file. Doing that would entail having setuptools actually start installing a copy of distutils into site-packages: older CPython releases would ignore it by default (since the stdlib version would shadow it), while 3.7+ would offer either "python3 -m ensuredistutils" or "python3 -m ensuresetuptools" (bikeshed to be painted via the PEP process) to install the setuptools-provided version.
I'm not claiming actually doing that would be particularly easy - I just think it's the most viable path to get us away from the current version coupling between the build infrastructure in distutils and the runtime support infrastructure in the rest of the standard library, and to avoid maintaining two distinct copies of distutils indefinitely (one in the stdlib, one in setuptools).
That approach wouldn't even entail any *new* bundling at the CPython level, as while it's currently formally an implementation detail (pending potential removal in a post-PEP-517 world), setuptools is already bundled as part of the support infrastructure for ensurepip: https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
I use distutils to build a C program that embeds Python, for making standalone launchers. I'm not at all clear how I'd get all the information I needed to build a compatible Python (including the "find the appropriate C compiler" dance) manually - I'd probably end up with something that didn't work except with specific Python versions or setups. Having said that, I assume that the relevant APIs or equivalents would be exposed via a future setuptools that included distutils (as it would need them internally to build extensions). Alternatively, I'd support refactoring these aspects of distutils out into a standalone library. Paul On 28 September 2017 at 15:55, Daniel Holth <dholth@gmail.com> wrote:
Enscons uses parts of distutils to get compiler flags and so on but does not use Extension() to do the actual compiling. There might be a cleaner way to do it that I was not able to find. There could be a cleaner separation between parts of distutils related to how Python itself was compiled and the part that does the actual compiling. The sysconfig module attempts to do at least part of this.
distutils also knows about some compiler tricks like generating symbol tables for Windows, that are not easy to re-use or invoke separately. Someone could catalog these tricks and techniques and refactor the useful ones into a reusable library. I don't have a great sense of how to do it.
On Wed, Sep 27, 2017 at 11:36 PM Nick Coghlan <ncoghlan@gmail.com> wrote:
On 28 September 2017 at 06:00, xoviat <xoviat@gmail.com> wrote:
That's actually an interesting idea though: for Python 3.7 distutils -> _distutils (and then setuptools is required for building). For/against?
distutils works fine for its original purpose (building components for the system Python in Linux distros), so we still need to avoid breaking that. setuptools is only essential if you want full support for modern *Python* level packaging features (PEP 376 install metadata, venv compatibility, wheel files, etc), and a lot of Linux system components simply don't worry about those things, and rely on their system level equivalents instead (e.g. the RPM/deb databases, chroots and containers, RPM/deb files)
However, what *could* be interesting is a proposal to move distutils to the "ensurepip" model, where rather than maintaining distutils directly as part of CPython, the CPython build process instead runs setuptools/distutils from a bundled wheel file. Doing that would entail having setuptools actually start installing a copy of distutils into site-packages: older CPython releases would ignore it by default (since the stdlib version would shadow it), while 3.7+ would offer either "python3 -m ensuredistutils" or "python3 -m ensuresetuptools" (bikeshed to be painted via the PEP process) to install the setuptools-provided version.
I'm not claiming actually doing that would be particularly easy - I just think it's the most viable path to get us away from the current version coupling between the build infrastructure in distutils and the runtime support infrastructure in the rest of the standard library, and to avoid maintaining two distinct copies of distutils indefinitely (one in the stdlib, one in setuptools).
That approach wouldn't even entail any *new* bundling at the CPython level, as while it's currently formally an implementation detail (pending potential removal in a post-PEP-517 world), setuptools is already bundled as part of the support infrastructure for ensurepip: https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
I meant to include a link, sorry. https://github.com/pfmoore/pylaunch/blob/master/build_zastub.py On 28 September 2017 at 16:11, Paul Moore <p.f.moore@gmail.com> wrote:
I use distutils to build a C program that embeds Python, for making standalone launchers. I'm not at all clear how I'd get all the information I needed to build a compatible Python (including the "find the appropriate C compiler" dance) manually - I'd probably end up with something that didn't work except with specific Python versions or setups.
Having said that, I assume that the relevant APIs or equivalents would be exposed via a future setuptools that included distutils (as it would need them internally to build extensions). Alternatively, I'd support refactoring these aspects of distutils out into a standalone library.
Paul
On 28 September 2017 at 15:55, Daniel Holth <dholth@gmail.com> wrote:
Enscons uses parts of distutils to get compiler flags and so on but does not use Extension() to do the actual compiling. There might be a cleaner way to do it that I was not able to find. There could be a cleaner separation between parts of distutils related to how Python itself was compiled and the part that does the actual compiling. The sysconfig module attempts to do at least part of this.
distutils also knows about some compiler tricks like generating symbol tables for Windows, that are not easy to re-use or invoke separately. Someone could catalog these tricks and techniques and refactor the useful ones into a reusable library. I don't have a great sense of how to do it.
On Wed, Sep 27, 2017 at 11:36 PM Nick Coghlan <ncoghlan@gmail.com> wrote:
On 28 September 2017 at 06:00, xoviat <xoviat@gmail.com> wrote:
That's actually an interesting idea though: for Python 3.7 distutils -> _distutils (and then setuptools is required for building). For/against?
distutils works fine for its original purpose (building components for the system Python in Linux distros), so we still need to avoid breaking that. setuptools is only essential if you want full support for modern *Python* level packaging features (PEP 376 install metadata, venv compatibility, wheel files, etc), and a lot of Linux system components simply don't worry about those things, and rely on their system level equivalents instead (e.g. the RPM/deb databases, chroots and containers, RPM/deb files)
However, what *could* be interesting is a proposal to move distutils to the "ensurepip" model, where rather than maintaining distutils directly as part of CPython, the CPython build process instead runs setuptools/distutils from a bundled wheel file. Doing that would entail having setuptools actually start installing a copy of distutils into site-packages: older CPython releases would ignore it by default (since the stdlib version would shadow it), while 3.7+ would offer either "python3 -m ensuredistutils" or "python3 -m ensuresetuptools" (bikeshed to be painted via the PEP process) to install the setuptools-provided version.
I'm not claiming actually doing that would be particularly easy - I just think it's the most viable path to get us away from the current version coupling between the build infrastructure in distutils and the runtime support infrastructure in the rest of the standard library, and to avoid maintaining two distinct copies of distutils indefinitely (one in the stdlib, one in setuptools).
That approach wouldn't even entail any *new* bundling at the CPython level, as while it's currently formally an implementation detail (pending potential removal in a post-PEP-517 world), setuptools is already bundled as part of the support infrastructure for ensurepip: https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
distutils works fine for its original purpose (building components for the system Python in Linux distros),
What does Linux have to with it? In the eagle days, I found it most helpful for Windows, actually. And it's very helpful for OS-X as well. It was also great for pure python bundling and installing. But regardless of original intent, it's clear that distutils has not turned out to be as cleanly extendible as is needed. Setuptools has its issues as well -- personally it has always driven me crazy with its mingling of run-time vs build-time vs install time functionality. I think it would be helpful to be explicit about the final goals before talking about whether and how to fork distutils. What I'd love to see is a build tool that is compatible with the "modern" packaging paradigm-- pip, pypi, wheels. ( and other systems like rpm, conda, ....) Essentially setuptools without the cruft. ( no easy install, pkg_resources, ...) This could be maintained as a third party package, like the current setuptools. If/when it becomes ubiquitous, an "ensure" style module could be added to the stdlib. So how do we get there? I would think a fork and merge of both setuptools and distutils would make sense. Preserve the building part of the API, so folks could simply replace the import line an their setup.py files to use the new tool. Distutils and setuptools remain in maintainable only mode for probably a long time :-( -CHB
so we still need to avoid breaking that. setuptools is only essential if you want full support for modern *Python* level packaging features (PEP 376 install metadata, venv compatibility, wheel files, etc), and a lot of Linux system components simply don't worry about those things, and rely on their system level equivalents instead (e.g. the RPM/deb databases, chroots and containers, RPM/deb files)
However, what *could* be interesting is a proposal to move distutils to the "ensurepip" model, where rather than maintaining distutils directly as part of CPython, the CPython build process instead runs setuptools/distutils from a bundled wheel file. Doing that would entail having setuptools actually start installing a copy of distutils into site-packages: older CPython releases would ignore it by default (since the stdlib version would shadow it), while 3.7+ would offer either "python3 -m ensuredistutils" or "python3 -m ensuresetuptools" (bikeshed to be painted via the PEP process) to install the setuptools-provided version.
I'm not claiming actually doing that would be particularly easy - I just think it's the most viable path to get us away from the current version coupling between the build infrastructure in distutils and the runtime support infrastructure in the rest of the standard library, and to avoid maintaining two distinct copies of distutils indefinitely (one in the stdlib, one in setuptools).
That approach wouldn't even entail any *new* bundling at the CPython level, as while it's currently formally an implementation detail (pending potential removal in a post-PEP-517 world), setuptools is already bundled as part of the support infrastructure for ensurepip: https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
So how do we get there? I would think a fork and merge of both setuptools and distutils would make sense. Preserve the building part of the API, so folks could simply replace the import line an their setup.py files to use the new tool.
There's no need to even replace the line in setup.py. Setuptools currently has the full capability to override the distutils namespace when it's imported first. I think if we extract distutils into setuptools, it's not difficult to list setuptools in install_requires or otherwise on Python 3.7 and above. And I think it will eventually be better for setuptools because the monkeypatch can be avoided. This might even solve the bootstrapping problem. 2017-09-28 11:08 GMT-05:00 Chris Barker - NOAA Federal < chris.barker@noaa.gov>:
distutils works fine for its original purpose (building components for the system Python in Linux distros),
What does Linux have to with it? In the eagle days, I found it most helpful for Windows, actually. And it's very helpful for OS-X as well.
It was also great for pure python bundling and installing.
But regardless of original intent, it's clear that distutils has not turned out to be as cleanly extendible as is needed.
Setuptools has its issues as well -- personally it has always driven me crazy with its mingling of run-time vs build-time vs install time functionality.
I think it would be helpful to be explicit about the final goals before talking about whether and how to fork distutils.
What I'd love to see is a build tool that is compatible with the "modern" packaging paradigm-- pip, pypi, wheels. ( and other systems like rpm, conda, ....)
Essentially setuptools without the cruft. ( no easy install, pkg_resources, ...)
This could be maintained as a third party package, like the current setuptools. If/when it becomes ubiquitous, an "ensure" style module could be added to the stdlib.
So how do we get there? I would think a fork and merge of both setuptools and distutils would make sense. Preserve the building part of the API, so folks could simply replace the import line an their setup.py files to use the new tool.
Distutils and setuptools remain in maintainable only mode for probably a long time :-(
-CHB
so we still need to avoid breaking that. setuptools is only essential if you want full support for modern *Python* level packaging features (PEP 376 install metadata, venv compatibility, wheel files, etc), and a lot of Linux system components simply don't worry about those things, and rely on their system level equivalents instead (e.g. the RPM/deb databases, chroots and containers, RPM/deb files)
However, what *could* be interesting is a proposal to move distutils to the "ensurepip" model, where rather than maintaining distutils directly as part of CPython, the CPython build process instead runs setuptools/distutils from a bundled wheel file. Doing that would entail having setuptools actually start installing a copy of distutils into site-packages: older CPython releases would ignore it by default (since the stdlib version would shadow it), while 3.7+ would offer either "python3 -m ensuredistutils" or "python3 -m ensuresetuptools" (bikeshed to be painted via the PEP process) to install the setuptools-provided version.
I'm not claiming actually doing that would be particularly easy - I just think it's the most viable path to get us away from the current version coupling between the build infrastructure in distutils and the runtime support infrastructure in the rest of the standard library, and to avoid maintaining two distinct copies of distutils indefinitely (one in the stdlib, one in setuptools).
That approach wouldn't even entail any *new* bundling at the CPython level, as while it's currently formally an implementation detail (pending potential removal in a post-PEP-517 world), setuptools is already bundled as part of the support infrastructure for ensurepip: https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Thu, Sep 28, 2017 at 8:46 AM, xoviat <xoviat@gmail.com> wrote:
No. Setuptools is what projects without a build_backend in pyproject.toml get. Not distutils. We should make it clear now that the distutils namespace belongs to setuptools except for when building cpython.
On Sep 27, 2017 2:33 PM, "Ned Deily" <nad@python.org> wrote:
On Sep 27, 2017, at 13:31, Steve Dower <steve.dower@python.org> wrote:
setuptools is totally welcome in my book to simply copy the compiler infrastructure we already have from core and never look back. It really does need to be maintained separately from CPython, especially on Windows where we continue to get innovation in the targeted tools. I know it's a big ask, and it's one that I can't personally commit real time to (though I obviously will as much as possible), but I do think it is necessary for our ecosystem to not be tied to CPython release cycles.
Whatever is done, keep in mind that currently distutils is required to build Python itself, e.g. the standard library. And that at least one important project, numpy, already subclasses distutils.
For numpy that seems fixable (if it even breaks, it may not). As long as the setuptools maintainers are willing to keep numpy.distutils compatibility, I'm happy to make the necessary changes in numpy. FYI, it has happened twice (IIRC) in the last five years that a new setuptools release broke numpy.distutils. This was fixed very quickly with good collaboration between the projects. Ralf
-- Ned Deily nad@python.org -- []
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
participants (11)
-
Brett Cannon -
Chris Barker - NOAA Federal -
Daniel Holth -
Donald Stufft -
Matthias Klose -
Ned Deily -
Nick Coghlan -
Paul Moore -
Ralf Gommers -
Steve Dower -
xoviat