PEP 517: Bootstrapping setuptools
Previously, the attempt to move setuptools off of vendored dependencies failed because it was not done correctly: install_requires was set to the vendored dependencies but not setup_requires, which would have been required to correctly specify dependencies. However, setup_requires would have introduced circular dependencies that are impossible to correctly resolve, so that would have simply shifted an impossible problem onto pip. The principle issue then, is that setuptools requires setuptools to build itself. And although, this problem was previously not worth solving because of 'setup.py install', it is now not a difficult problem to solve if we rethink slightly what should be included in the setuptools respository: - Rather than re-generating egg_info on each setup.py invocation, we can cache dist-info in the setuptools respository. - We can implement a simple entry_points script that updates dist-info based on the contents of setup.py, which is only runnable of course after setuptools is installed - We can implement the reference build backend for setuptools: simply copy the files and dist-info into a new compliant zip archive Viola! Setuptools is no longer required to build setuptools, and the installation process is fully compliant with Python PEPs.
Of course, to be frank, the principle failure with this plan is that third-party tools (buildout, os packagers) will not be compliant with PEP 517 even after it is adopted, and will then complain about having to update their build systems. 2017-08-21 16:05 GMT-05:00 xoviat <xoviat@gmail.com>:
Previously, the attempt to move setuptools off of vendored dependencies failed because it was not done correctly: install_requires was set to the vendored dependencies but not setup_requires, which would have been required to correctly specify dependencies. However, setup_requires would have introduced circular dependencies that are impossible to correctly resolve, so that would have simply shifted an impossible problem onto pip.
The principle issue then, is that setuptools requires setuptools to build itself. And although, this problem was previously not worth solving because of 'setup.py install', it is now not a difficult problem to solve if we rethink slightly what should be included in the setuptools respository: - Rather than re-generating egg_info on each setup.py invocation, we can cache dist-info in the setuptools respository. - We can implement a simple entry_points script that updates dist-info based on the contents of setup.py, which is only runnable of course after setuptools is installed - We can implement the reference build backend for setuptools: simply copy the files and dist-info into a new compliant zip archive
Viola! Setuptools is no longer required to build setuptools, and the installation process is fully compliant with Python PEPs.
On Mon, Aug 21, 2017 at 5:17 PM, xoviat <xoviat@gmail.com> wrote:
Of course, to be frank, the principle failure with this plan is that third-party tools (buildout, os packagers) will not be compliant with PEP 517 even after it is adopted, and will then complain about having to update their build systems.
I don't understand this statement. If buildout needs to be compliant, it will be (eventually). I can imagine a future where everything is distributed with wheels and buildout wouldn't need to build anything. Jim
2017-08-21 16:05 GMT-05:00 xoviat <xoviat@gmail.com>:
Previously, the attempt to move setuptools off of vendored dependencies failed because it was not done correctly: install_requires was set to the vendored dependencies but not setup_requires, which would have been required to correctly specify dependencies. However, setup_requires would have introduced circular dependencies that are impossible to correctly resolve, so that would have simply shifted an impossible problem onto pip.
The principle issue then, is that setuptools requires setuptools to build itself. And although, this problem was previously not worth solving because of 'setup.py install', it is now not a difficult problem to solve if we rethink slightly what should be included in the setuptools respository: - Rather than re-generating egg_info on each setup.py invocation, we can cache dist-info in the setuptools respository. - We can implement a simple entry_points script that updates dist-info based on the contents of setup.py, which is only runnable of course after setuptools is installed - We can implement the reference build backend for setuptools: simply copy the files and dist-info into a new compliant zip archive
Viola! Setuptools is no longer required to build setuptools, and the installation process is fully compliant with Python PEPs.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Jim Fulton http://jimfulton.info
The unvendoring means that setuptools and the project code are now competing over who gets to define what an acceptable version is for these
This statement comes from something that Donald said: libraries to be installed with. As if this isn't going to be true for any other build system, and setuptools won't at all be in any way special in a PEP 517 world. So you are always going to have complaints about "well build system X required a newer version but my project requires an older version" when the real problem is that people are requiring an older version/ projects do not have a stable API. 2017-08-21 16:27 GMT-05:00 Jim Fulton <jim@jimfulton.info>:
On Mon, Aug 21, 2017 at 5:17 PM, xoviat <xoviat@gmail.com> wrote:
Of course, to be frank, the principle failure with this plan is that third-party tools (buildout, os packagers) will not be compliant with PEP 517 even after it is adopted, and will then complain about having to update their build systems.
I don't understand this statement. If buildout needs to be compliant, it will be (eventually). I can imagine a future where everything is distributed with wheels and buildout wouldn't need to build anything.
Jim
2017-08-21 16:05 GMT-05:00 xoviat <xoviat@gmail.com>:
Previously, the attempt to move setuptools off of vendored dependencies failed because it was not done correctly: install_requires was set to the vendored dependencies but not setup_requires, which would have been required to correctly specify dependencies. However, setup_requires would have introduced circular dependencies that are impossible to correctly resolve, so that would have simply shifted an impossible problem onto pip.
The principle issue then, is that setuptools requires setuptools to build itself. And although, this problem was previously not worth solving because of 'setup.py install', it is now not a difficult problem to solve if we rethink slightly what should be included in the setuptools respository: - Rather than re-generating egg_info on each setup.py invocation, we can cache dist-info in the setuptools respository. - We can implement a simple entry_points script that updates dist-info based on the contents of setup.py, which is only runnable of course after setuptools is installed - We can implement the reference build backend for setuptools: simply copy the files and dist-info into a new compliant zip archive
Viola! Setuptools is no longer required to build setuptools, and the installation process is fully compliant with Python PEPs.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Jim Fulton http://jimfulton.info
On Mon, Aug 21, 2017 at 5:39 PM, xoviat <xoviat@gmail.com> wrote:
This statement comes from something that Donald said:
The unvendoring means that setuptools and the project code are now competing over who gets to define what an acceptable version is for these libraries to be installed with.
As if this isn't going to be true for any other build system, and setuptools won't at all be in any way special in a PEP 517 world. So you are always going to have complaints about "well build system X required a newer version but my project requires an older version" when the real problem is that people are requiring an older version/ projects do not have a stable API.
This boils down to implementation details. <shrug> pip dealt with this issue and buildout stumbled because of differences in the way they managed their paths. Buildout could have coped (eventually). Jim
2017-08-21 16:27 GMT-05:00 Jim Fulton <jim@jimfulton.info>:
On Mon, Aug 21, 2017 at 5:17 PM, xoviat <xoviat@gmail.com> wrote:
Of course, to be frank, the principle failure with this plan is that third-party tools (buildout, os packagers) will not be compliant with PEP 517 even after it is adopted, and will then complain about having to update their build systems.
I don't understand this statement. If buildout needs to be compliant, it will be (eventually). I can imagine a future where everything is distributed with wheels and buildout wouldn't need to build anything.
Jim
2017-08-21 16:05 GMT-05:00 xoviat <xoviat@gmail.com>:
Previously, the attempt to move setuptools off of vendored dependencies failed because it was not done correctly: install_requires was set to the vendored dependencies but not setup_requires, which would have been required to correctly specify dependencies. However, setup_requires would have introduced circular dependencies that are impossible to correctly resolve, so that would have simply shifted an impossible problem onto pip.
The principle issue then, is that setuptools requires setuptools to build itself. And although, this problem was previously not worth solving because of 'setup.py install', it is now not a difficult problem to solve if we rethink slightly what should be included in the setuptools respository: - Rather than re-generating egg_info on each setup.py invocation, we can cache dist-info in the setuptools respository. - We can implement a simple entry_points script that updates dist-info based on the contents of setup.py, which is only runnable of course after setuptools is installed - We can implement the reference build backend for setuptools: simply copy the files and dist-info into a new compliant zip archive
Viola! Setuptools is no longer required to build setuptools, and the installation process is fully compliant with Python PEPs.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Jim Fulton http://jimfulton.info
-- Jim Fulton http://jimfulton.info
Isn't this a special case of needing . on sys.path when building a package? Then the same version of setuptools that is being packaged builds itself. I've experimented with a different kind of bootstrapping where pip installs setuptools on demand when the first setuptools requiring package is built. You kindof need pkg-resources as a separate package for this. It is a cool parlor trick but it's simpler and harmless to just always install setuptools like we do. On Tue, Aug 22, 2017, 09:14 Jim Fulton <jim@jimfulton.info> wrote:
On Mon, Aug 21, 2017 at 5:39 PM, xoviat <xoviat@gmail.com> wrote:
This statement comes from something that Donald said:
The unvendoring means that setuptools and the project code are now competing over who gets to define what an acceptable version is for these libraries to be installed with.
As if this isn't going to be true for any other build system, and setuptools won't at all be in any way special in a PEP 517 world. So you are always going to have complaints about "well build system X required a newer version but my project requires an older version" when the real problem is that people are requiring an older version/ projects do not have a stable API.
This boils down to implementation details. <shrug> pip dealt with this issue and buildout stumbled because of differences in the way they managed their paths. Buildout could have coped (eventually).
Jim
2017-08-21 16:27 GMT-05:00 Jim Fulton <jim@jimfulton.info>:
On Mon, Aug 21, 2017 at 5:17 PM, xoviat <xoviat@gmail.com> wrote:
Of course, to be frank, the principle failure with this plan is that third-party tools (buildout, os packagers) will not be compliant with PEP 517 even after it is adopted, and will then complain about having to update their build systems.
I don't understand this statement. If buildout needs to be compliant, it will be (eventually). I can imagine a future where everything is distributed with wheels and buildout wouldn't need to build anything.
Jim
2017-08-21 16:05 GMT-05:00 xoviat <xoviat@gmail.com>:
Previously, the attempt to move setuptools off of vendored dependencies failed because it was not done correctly: install_requires was set to the vendored dependencies but not setup_requires, which would have been required to correctly specify dependencies. However, setup_requires would have introduced circular dependencies that are impossible to correctly resolve, so that would have simply shifted an impossible problem onto pip.
The principle issue then, is that setuptools requires setuptools to build itself. And although, this problem was previously not worth solving because of 'setup.py install', it is now not a difficult problem to solve if we rethink slightly what should be included in the setuptools respository: - Rather than re-generating egg_info on each setup.py invocation, we can cache dist-info in the setuptools respository. - We can implement a simple entry_points script that updates dist-info based on the contents of setup.py, which is only runnable of course after setuptools is installed - We can implement the reference build backend for setuptools: simply copy the files and dist-info into a new compliant zip archive
Viola! Setuptools is no longer required to build setuptools, and the installation process is fully compliant with Python PEPs.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Jim Fulton http://jimfulton.info
-- Jim Fulton http://jimfulton.info _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Tue, Aug 22, 2017 at 9:23 AM, Daniel Holth <dholth@gmail.com> wrote:
Isn't this a special case of needing . on sys.path when building a package? Then the same version of setuptools that is being packaged builds itself.
No. The issue for us it wasn't setuptools itself, but it's dependencies and those dependencies conflicted with dependencies of of packages we were installing *and* those packages importing these dependences (indirectly) in their setup scripts. Setup scripts that import the thing they're about to install, generally to get the version :(, is something I'd love to see go away. Jim -- Jim Fulton http://jimfulton.info
venvs within venvs... terrifying concept. On Tue, Aug 22, 2017 at 11:02 AM, Jim Fulton <jim@jimfulton.info> wrote:
On Tue, Aug 22, 2017 at 9:23 AM, Daniel Holth <dholth@gmail.com> wrote:
Isn't this a special case of needing . on sys.path when building a package? Then the same version of setuptools that is being packaged builds itself.
No. The issue for us it wasn't setuptools itself, but it's dependencies and those dependencies conflicted with dependencies of of packages we were installing *and* those packages importing these dependences (indirectly) in their setup scripts. Setup scripts that import the thing they're about to install, generally to get the version :(, is something I'd love to see go away.
Jim
-- Jim Fulton http://jimfulton.info
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
I didn't mention (nor do I recall anyone mentioning) venvs. Jim On Tue, Aug 22, 2017 at 11:15 AM, Matt Joyce <matt@nycresistor.com> wrote:
venvs within venvs... terrifying concept.
On Tue, Aug 22, 2017 at 11:02 AM, Jim Fulton <jim@jimfulton.info> wrote:
On Tue, Aug 22, 2017 at 9:23 AM, Daniel Holth <dholth@gmail.com> wrote:
Isn't this a special case of needing . on sys.path when building a package? Then the same version of setuptools that is being packaged builds itself.
No. The issue for us it wasn't setuptools itself, but it's dependencies and those dependencies conflicted with dependencies of of packages we were installing *and* those packages importing these dependences (indirectly) in their setup scripts. Setup scripts that import the thing they're about to install, generally to get the version :(, is something I'd love to see go away.
Jim
-- Jim Fulton http://jimfulton.info
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Jim Fulton http://jimfulton.info
So are we going for a "primarily isolation" approach then where builds are only isolated on the first level, but subsequent levels share the same build environment? 2017-08-22 10:23 GMT-05:00 Jim Fulton <jim@jimfulton.info>:
I didn't mention (nor do I recall anyone mentioning) venvs.
Jim
On Tue, Aug 22, 2017 at 11:15 AM, Matt Joyce <matt@nycresistor.com> wrote:
venvs within venvs... terrifying concept.
On Tue, Aug 22, 2017 at 11:02 AM, Jim Fulton <jim@jimfulton.info> wrote:
On Tue, Aug 22, 2017 at 9:23 AM, Daniel Holth <dholth@gmail.com> wrote:
Isn't this a special case of needing . on sys.path when building a package? Then the same version of setuptools that is being packaged builds itself.
No. The issue for us it wasn't setuptools itself, but it's dependencies and those dependencies conflicted with dependencies of of packages we were installing *and* those packages importing these dependences (indirectly) in their setup scripts. Setup scripts that import the thing they're about to install, generally to get the version :(, is something I'd love to see go away.
Jim
-- Jim Fulton http://jimfulton.info
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Jim Fulton http://jimfulton.info
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Oh, gawd. From now on, someone will have to say buildout 3 times before I appear. On Tue, Aug 22, 2017 at 2:06 PM, xoviat <xoviat@gmail.com> wrote:
So are we going for a "primarily isolation" approach then where builds are only isolated on the first level, but subsequent levels share the same build environment?
I'm a huge fan of isolation, but isolation seems to me to be orthogonal to this discussion. I'm done with this thread. :) Jim
2017-08-22 10:23 GMT-05:00 Jim Fulton <jim@jimfulton.info>:
I didn't mention (nor do I recall anyone mentioning) venvs.
Jim
On Tue, Aug 22, 2017 at 11:15 AM, Matt Joyce <matt@nycresistor.com> wrote:
venvs within venvs... terrifying concept.
On Tue, Aug 22, 2017 at 11:02 AM, Jim Fulton <jim@jimfulton.info> wrote:
On Tue, Aug 22, 2017 at 9:23 AM, Daniel Holth <dholth@gmail.com> wrote:
Isn't this a special case of needing . on sys.path when building a package? Then the same version of setuptools that is being packaged builds itself.
No. The issue for us it wasn't setuptools itself, but it's dependencies and those dependencies conflicted with dependencies of of packages we were installing *and* those packages importing these dependences (indirectly) in their setup scripts. Setup scripts that import the thing they're about to install, generally to get the version :(, is something I'd love to see go away.
Jim
-- Jim Fulton http://jimfulton.info
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Jim Fulton http://jimfulton.info
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Jim Fulton http://jimfulton.info
The OP was about "bootstrapping setuptools." If, as Leonardo said, we only isolate on the first level (requirements from install_requires or user will be isolated but requirements from build_requires will share the same environment as the invoker of build_requires if they are not wheels), then that opens up some interesting possibilities. However, I don't necessarily think that this is a good idea because it opens up situations that could be nasty to debug. I still think the simpler method would be as I have said previously, to have a very minimal backend that doesn't depend on anything and simply archives the files and info. We can even put this backend into a separate package that requires and depends on nothing, so that future build systems don't have to re-invent the wheel here. The question that I have is: how does flit address this issue? Does flit require external dependencies to bootstrap itself? 2017-08-22 13:50 GMT-05:00 Jim Fulton <jim@jimfulton.info>:
Oh, gawd. From now on, someone will have to say buildout 3 times before I appear.
On Tue, Aug 22, 2017 at 2:06 PM, xoviat <xoviat@gmail.com> wrote:
So are we going for a "primarily isolation" approach then where builds are only isolated on the first level, but subsequent levels share the same build environment?
I'm a huge fan of isolation, but isolation seems to me to be orthogonal to this discussion.
I'm done with this thread. :)
Jim
2017-08-22 10:23 GMT-05:00 Jim Fulton <jim@jimfulton.info>:
I didn't mention (nor do I recall anyone mentioning) venvs.
Jim
On Tue, Aug 22, 2017 at 11:15 AM, Matt Joyce <matt@nycresistor.com> wrote:
venvs within venvs... terrifying concept.
On Tue, Aug 22, 2017 at 11:02 AM, Jim Fulton <jim@jimfulton.info> wrote:
On Tue, Aug 22, 2017 at 9:23 AM, Daniel Holth <dholth@gmail.com> wrote:
Isn't this a special case of needing . on sys.path when building a package? Then the same version of setuptools that is being packaged builds itself.
No. The issue for us it wasn't setuptools itself, but it's dependencies and those dependencies conflicted with dependencies of of packages we were installing *and* those packages importing these dependences (indirectly) in their setup scripts. Setup scripts that import the thing they're about to install, generally to get the version :(, is something I'd love to see go away.
Jim
-- Jim Fulton http://jimfulton.info
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Jim Fulton http://jimfulton.info
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Jim Fulton http://jimfulton.info
On 23 August 2017 at 05:24, xoviat <xoviat@gmail.com> wrote:
The question that I have is: how does flit address this issue? Does flit require external dependencies to bootstrap itself?
It does, but those dependencies currently use setuptools as their packaging system rather than flit. Bootstrapping self-hosted packaging tools is a genuinely hard problem - it's why pip vendors most of its dependencies, and relies on preinstalled SSL/TLS support in the standard library. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Aug 23, 2017, at 05:37 AM, Nick Coghlan wrote:
On 23 August 2017 at 05:24, xoviat <xoviat@gmail.com> wrote:
The question that I have is: how does flit address this issue? Does flit require external dependencies to bootstrap itself?
It does, but those dependencies currently use setuptools as their packaging system rather than flit.
Yup. I actually moved a package which I maintain (zipfile36) back to setuptools packaging specifically so that flit could use it without a dependency loop. The other dependencies for bootstrapping are requests and docutils. There's one other dependency listed, but that can be ignored for bootstrapping. Thomas
On 23 August 2017 at 01:37, Nick Coghlan <ncoghlan@gmail.com> wrote:
[...]
Bootstrapping self-hosted packaging tools is a genuinely hard problem
True, but on the other hand, it's the self-hosted packaging tools' own problem. It's up to the tools themselves to solve it. if PEP 517 simply establishes that the current directory of the source distribution will be naturally at the head of `sys.path` (whether an original checkout or a cleaned up sdist), then self-hosted packaging tools can be their own backend, and do whatever magick they need to self-bootstrap. It makes sense to think a bit about whether we're not making these cases harder, but we shouldn't block progress of PEP 517 on account of these cases. If we specifically want to make these cases easier in the future, we might think of another PEP to expose to backends certain frontend operations like "provide me the sdist (or wheel) for requirement foo >= 2.3.0 but don't install it and don't fetch dependencies" so we can make sure certain configurations like "what is the index server?" or "how do I use a web proxy?" are centralized and consistent. Regards, Leo
Hi, On Mon, Aug 21, 2017 at 5:39 PM, xoviat <xoviat@gmail.com> wrote:
This statement comes from something that Donald said:
The unvendoring means that setuptools and the project code are now competing over who gets to define what an acceptable version is for these libraries to be installed with.
As if this isn't going to be true for any other build system, and setuptools won't at all be in any way special in a PEP 517 world. So you are always going to have complaints about "well build system X required a newer version but my project requires an older version" when the real problem is that people are requiring an older version/ projects do not have a stable API.
It seems to me that this is one of those things where the solution is staring us in the face. If PEP 517 simply clarifies that, since the build steps should be run with either the original checkout or a cleaned-up sdist as the current directory, this directory will be naturally the first on `sys.path` (and shouldn't be removed), so build backends can rely on its presence there. In `setuptools` case, if it wants to depend at build time (i.e. `setup_requires`) on other packages that depend on `setuptools', it can't specify them as `[build-system]requires`, but it's free to specify them as `get_requires_for_build*()`, since `setuptools` itself will be on `sys.path`. `get_requires_for_build*()` just needs to arrange for the source distribution directory to contain enough of whatever metadata format is needed to be recognized as an installed dist to be depended upon while being on sys.path. As mentioned, the real trouble begins when `setuptools` wants to depend at runtime (`install_requires`) on other packages since incompatible versions of these packages could spell trouble to packages that depend on setuptools AND on these incompatible versions. If setuptools doesn't want to "hard vendor" these dependencies (i.e. keep a copy of them in its own repository), one possible solution is for `setuptools` to vendor these dependencies at build time: - make them build dependencies as above - locate where they were installed by their `.dist-info/RECORD`, or failing that, their `.__file__` module attribute. - copy them over to it's own `setuptools._vendor` package, patching them if necessary. An enviroment (e.g. upstream distributor) wanting to bootstrap everything from source would then need to make sure setuptools and all its dependencies are installed in the same python environment in "development mode" (/me handwaves the fact that this operation is not strictly defined). A frontend could be written to do just that. Regards, Leo
participants (8)
-
Daniel Holth -
Jim Fulton -
Leonardo Rochael Almeida -
Matt Joyce -
Matthias Klose -
Nick Coghlan -
Thomas Kluyver -
xoviat